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SUMMARY 


The  azaad  services  h ave  a  goal  of  adalnlsterlng  thalr  op« rational 
selection  and  claaalflcatlon  anil a tad  test,  tha  Araad  Services  Vocational 
Aptitude  Bat tar y  (ASVAB) ,  on  tha  conputar.  Intent Iona  are  to  adnlnlatar 
tha  aaaa  kind  of  anbtaata  aa  axlat  in  tha  praaant  operational  papar-and- 
pancil  (PAP)  battery  and  to  be  able  to  do  ao  in  ouch  a  aanaer  aa  to  naka  it 
totally  Irrelevant  whether  an  axaalnee  receives  tha  coaputar-adalnlstered 
anbtaat  or  tha  PAP  vara ion. 

Several  anbtaata  conalat  of  ltaaa  which  by  thalr  vary  nature  would 
aaaa  to  be  potential  probleaa  for  tranafar  to  tha  cathode-ray  tuba  (CRT)* 
Thera  are  two  different  typea  of  apaadad  anbtaata  (nuaarlcal  operations  and 
coding  spaed),  whose  ltaaa  are  easy  and  on  which  tha  final  score  generally 
depends  on  the  nuaber  of  ltaaa  accoapllshad  in  a  short  tins  Halt.  The  CRT 
is  such  too  saall  to  duplicate  a  full  page  of  ltaaa  such  as  those  presented 
in  a  PAP  aoda.  Similarly,  tha  alas  of  paragraph  coaprehenslon  lteas 
prevents  thalr  appearing  fully  on  tha  CRT  screen.  Finally,  three  of  the 
ASVAB  subtests  have  lteas  with  illustrations,  and  CRT  presentation  oust 
switch  froa  flat  ink  drawings  to  vertical  light  drawings  on  the  screen. 

Speeded  subteats  ware  prograaaad  for  two  different  CRT  presentation 
node a,  paragraph  coaprehenslon  for  three  different  nodes,  and  graphics  were 
displayed  froa  code  created  by  an  off-the-shelf  coaaerclal  digitiser. 

These  CRT  aubtests  were  then  coapared  with  their  PAP  counterparts  using  Air 
Force  recruits  in  a  counterbalanced  design.  Results  indicated  that 
■obtaining  equivalence  between  PAP  and  eoaputer  administration  appears 
feasible.  Graphical  ltaaa  present  the  least  difficulty.  .  Itaa  production 
for  one  type  of  speeded  aubtest  was  beat  approxlaated  by  slngle-ltea  CRT 
presentation,  and  for  another  type  of  apeeded  teat  by  CRT  presentation  of 
blocks  of  ltaaa.  Additional  research  is  required  for  the  "too  long" 
paragraph  coaprehenslon  lteas  in  which  wore  practice  with  the  eoaputer 
scope  aay  be  useful. 


PREFACE 


Several  Aratd  Services  Vocational  Aptitude  Battery  (ASVAB)  subtests 
consist  of  itens  whose  characteristics  were  expected  to  change  when 
edninistered  on  e  computer  screen.  This  technical  paper  exanlnes  several 
ways  to  edninister  these  itens  in  order  to  find  one  which  is  equivalent  to 
paper-and-pencll  adninis tret ion.  This  effort  is  ancillary  to  the  Air  Force 
responsibility  for  itea  pools  for  e  conputer  adapted  ASVAB. 
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I.  INTRODUCTION 

As  the  United  States  Armed  Service!  move  toward  the  iapleaentation  of 
computer  adaptive  tasting  (CAT)  with  the  Armed  Services  Vocational  Aptitude 
Battery  (ASVAB),  several  major  issues  oust  be  addresaad  to  ensure  the  continu¬ 
ity  and  well-being  of  the  ailitary  tasting  progran.  At  the  present  tine,  the 
transition  froa  traditional  papa r-end-penc il  testing  to  coaputarlaed  tasting 
is  being  envisioned  as  a  gradual  process;  therefore,  the  equivalence  of  the 
two  coexisting  aathods  of  testing  is  of  concern,  because  the  resulting  scores 
will  be  required  to  function  Interchangeably  (Green,  Bock,  Humphreys,  Linn,  & 
Recks se,  1982). 

The  first  question  of  equivalence  that  aust  be  addressed  la  that  of  the 
Influence  of  the  presentation  aadiua  itself  on  otherwise  identical  tests. 

That  is,  if  the  ASVAB  in  its  present  fora  is  adalnlstered  by  coaputar  to  a 
group  of  exaainees,  how  would  the  resulting  scores  differ  froa  those  that 
would  be  obtained  by  testing  in  the  paper-and-pencil  asdlua?  More  specifical¬ 
ly,  three  questions  can  be  raised: 

1.  How  would  the  scores  differ  by  subtest? 

2.  What  night  account  for  aoas  of  these  differences? 

3.  For  sons  types  of  itaaa,  which  of  several  nodes  of  presentation  on  the 
coaputar  screen  night  serve  to  ainiaiaa  differences? 

The  present  effort  was  designed  to  address  these  questions  in  two  ways. 
One  way  that  the  equivalence  of  two  procedures  can  be  dateralned  is  by  a  com¬ 
parison  of  the  naan  scores  resulting  froa  then  to  see  if  significant  differ¬ 
ences  can  be  detected.  Toward  this  end,  tkn  experlaantal  design  provided  sep¬ 
arate  conditions  for  the  study  of  alternative  coaputar  procedures  against 
their  paper-end-pencil  counterparts.  In  addition,  controls  were  built  into 
the  study  to  account  for  possible  fora  and  version  differences,  as  well  as 
practice  effects.  Methods  of  coaparlng  assn  differences  were  then  applied. 

The  second  approach  was  that  of  aaploylng  correlational  aathods.  These 
techniques,  which  atteapt  to  assess  the  degree  of  slallarlty  rather  than  the 
extent  of  differences,  Included  test-retest  and  Internal  consistency  reliabil¬ 
ity  coaparlaons  across  conditions,  and  exploratory  and  confirmatory  factor 
analytic  studies  to  coapare  factor  structures. 

For  the  purposes  of  this  effort,  6  of  the  10  ASVAB  subtests  were  chosen 
for  administration.  The  selection  was  based  on  sons  characteristic  or  set  of 
characteristics  of  the  subtests  that  night  be  expected  to  Interact  with  the 
aedlua  of  presentation  in  either  a  positive  or  negative  way,  or  that  night 
require  sons  change  or  modification  froa  the  previous  paper-and-pencil  fora  of 
presentation.  These  subtests  can  be  divided  into  three  groups  on  the  basis  of 
the  criteria  for  selection:  Paragraph  Comprehension  (PC),  because  of  the 
problems  Involved  in  presenting  it  by  computer  that  result  froa  its  unique 
ltea  foraat;  Numerical  Operations  (NO)  and  Coding  Speed  (CS),  due  to  the 
speeded  nature  of  these  subtests;  and  Auto  and  Shop  Information  (AS),  Mechani¬ 
cal  Comprehension  (MC),  and  Electronics  Information  (El),  because  of  their 
eaphasls  on  graphical  iaages,  as  wall  as  Items  with  no  graphical  content. 
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II.  METHOD 


Test  Battery 

This  study  used  subtests  of  Fonts  11,  12,  end  13  of  the  ASVAB  (Prestwood, 
Vele,  Messey,  &  Welsh,  1985).  The  subtests  used  ere  shown  In  Teble  1.  The 
six  subtests  chosen  for  study  were  administered  in  two  versions,  A  end  B,  for 
each  of  the  three  forms  for  e  total  of  six  alternatives  for  each  subtest:  11A, 
11B,  12A,  12B,  13A,  and  13B.  For  the  CS,  AS,  MC,  end  El  subtests,  the  two 
versions  differ  only  in  the  ordering  of  the  items;  by  contrast,  the  PC  end  MO 
subtests  use  different  items  in  each  version.  All  subtests  consist  of  differ¬ 
ent  items  in  each  form.  Scoring  was  based  on  number-correct  raw  scores  for 
each  subtest. 


Table  1.  ASVAB  Subtests  Used  with  Corresponding  Number 
of  Items,  Operational  Tima  in  Minutes,  end  Speededness 


Subtest 

Number 
of  items 

Time 

allotted 

Speeded 

Paragraph  Comprehension  (PC) 

IS 

13 

No 

Numerical  Operations  (NO) 

50 

3 

Tea 

Coding  Speed  (CS) 

84 

7 

Yes 

Auto  end  Shop  Information  (AS) 

25 

11 

No 

Mechanical  Comprehension  (MC) 

25 

19 

No 

Electronics  Information  (El) 

20 

9 

No 

Experimental  Design 


Subjects  and  Pete  Collection 


The  initial  sample  consisted  of  1,024  Air  Force  recruits  distributed  over 
30  Independent  groups  with  repeated  measures  for  each  examinee.  The  30  groups 
were  subgroupa  of  three  general  experimental  groups:  (a)  Group  1,  which  was 
first  administered  the  ASVAB  tests  by  computer  and  then  by  paper-and-pencll; 
(b)  Group  2,  which  took  the  tests  first  by  paper-and-pencll  and  then  by  com¬ 
puter;  end  (c)  Group  3  which  took  the  ASVAB  tests  twice  by  paper-and-pencll. 

After  removal  of  27  examinees  with  incomplete  data,  the  final  date  con¬ 
sisted  of  item  responses  end  subtest  scores  for  997  Air  Force  recruits  on  six 
subtests  of  the  ASVAB,  each  taken  twice  in  alternate  forms,  for  e  total  of  12 
subtest  scores  per  examinee.  Randomization  was  assured  by  preprinting  1,200 
cards  with  coded  conditions  and  ordering  then  in  40  groups,  each  group  con¬ 
taining  conditions  1  through  30  in  sequence.  Examinees  were  then  assigned  to 
groups  by  having  than  line  up  upon  their  arrival  at  the  test  center  and  hand¬ 
ing  out  the  cards  in  order  until  ell  examinees  were  accounted  for.  In  this 
way,  every  30th  person  was  assigned  to  the  same  condition.  All  examinees  were 
male  recruits  (to  avoid  possible  confounding  of  results  due  to  gender  differ¬ 
ences),  and  ell  were  in  their  sixth  day  of  basic  military  training  at  the  time 
of  testing.  Testing  of  examinees  occurred  at  the  Air  Force  Human  Resources 
Laboratory  test  facility  at  Lackland  AFB,  Texas,  over  a  2-week  period  during 
February  1985. 
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One*  conditions  were  assigned,  Group  3  (the  group  receiving  papar-snd- 
poncll  tests  In  both  sessions)  was  renoved  to  a  separata  building  for  tasting 
and  was  not  exposed  to  the  conputerlsed  test  environment.  Group  1  was  brought 
Into  a  room  containing  30  carrels  with  Terak  nlcrocoaputers ,  28  of  which  were 
used  for  testing.  Examinees  were  assigned  randomly  to  computers.  Due  to  com¬ 
puter  memory  limitations,  each  computer  was  programmed  to  administer  only  ver¬ 
sion  A  or  version  B  tests.  Within  version,  however,  each  machine  contained 
all  forma  and  conditions  for  all  six  subtests.  The  test  each  examinee  re¬ 
ceived  was  determined  by  the  condition  code  printed  on  his  condition  card, 
which  was  entered  Into  the  computer  before  the  start  of  testing.  This  random 
assignment  of  examinees  to  conditions  and  to  computers  controlled  for  varia¬ 
tions  in  screen  resolution  between  computer  monitors — a  particularly  Important 
consideration  for  subtests  containing  graphical  Images. 

Before  testing  commenced,  Group  1  received  a  standard  set  of  Instruc¬ 
tions,  over  earphones  from  trained  test  administrators ,  covering  both  the 
tests  themselves  and  the  operation  of  the  computer;  then,  graphical  demonstra¬ 
tions  and  user-paced  exercises  on  computer  keyboard  use  and  the  10  keys  re¬ 
quired  for  testing  were  administered.  All  examinees  also  received  standard 
ASVAB  test  Instructions.  Administrators  ware  available  throughout  testing  to 
answer  questions  and  to  help  with  problene.  Upon  completion  of  testing,  exam¬ 
inees  were  allowed  to  leave  quietly.  During  the  first  session.  Group  2  was 
administered  equivalent  pape r-and-penc 11  tasts  In  another  room.  Following  a 
break  at  the  end  of  Sesalon  1,  Groups  1  and  2  switched  places;  l.e..  Group  2 
was  administered  instructions  and  tasting  by  computer  while  Group  1  was  admin¬ 
istered  paper-and-pendl  tests. 

Tests  and  Conditions 

PC  subtest.  This  subtest  poses  two  problems  when  converted  from  paper* 
end-pencil  to  computer  administration.  First,  sons  of  the  paragraphs  used  In 
the  paper-end-pencil  test  are  too  long  to  fit  on  a  cathode-ray-tube  (CRT) 
screen  at  one  tins.  Second,  most  of  the  reading  comprehension  paragraphs  In 
the  ASVAB  tests  are  accompanied  by  multiple  questions.  Consequently,  In  the 
computer  presentation  sodium,  three  different  methods  of  presenting  these 
kinds  of  Items  on  a  CRT  (explained  further  on  p.  4)  were  evaluated  to  deter¬ 
mine  which  gave  results  most  similar  to  those  of  pape  r-and-penc 11  administra¬ 
tion. 

MO  and  CS  subtests.  In  the  case  of  these  highly-speeded,  low-difficulty 
subtests.  Items  are  typically  presented  In  groups  In  the  paper-end-pencil  me¬ 
dium,  with  instructions  to  answer  as  nsny  Itsms  as  possible  within  the  time 
limit.  A  similar  approach  was  taken  with  the  computer  presentation  of  these 
subtests  by  presenting  several  Items  on  the  screen  at  a  time.  This  condition 
was  compared  with  a  second  condition  that  presented  Items  on  the  screen  one  at 
a  time,  to  determine  which  Item  presentation  condition  was  more  similar  to  the 
paper-and-pencll  adinlnlstratlon. 

AS.  MC.  and  El  subtests.  These  subtests  consist  of  both  standard  multi¬ 
ple-choice  test  Items  and  multiple-choice  items  that  use  graphical  images  to 
describe  physical,  mechanical,  or  electronic  concepts  or  components  about 
which  the  examinee  Is  questioned.  They  were  presented  on  the  computer  screen 
In  a  single  computer-presentation  node  very  similar  to  the  presentation  of 
their  paper-and-pencll  counterparts  ,  to  determine  if  differences  resulted  from 


the  translation  of  standard  multiple-choice  items  and  graphical  inagas  froa 
tbs  peper-and-pencll  aadiua  to  a  CRT. 


Traatnant  Gronpa 


Tha  overall  data  collection  plan,  with  tha  total  tmabars  of  axaalnaaa  par 
condition  (before  eliminating  those  with  incoaplete  data),  is  shown  in  Figures 
1  through  3.  Bach  figure  is  divided  into  sections  representing  the  treatment 
groups  to  which  tha  exaainees  were  readonly  assigned.  A  aajor  advantage  of 
this  design  was  that  it  allowed  the  conputer  versus  papa r-and-pencll  test  ad- 
ninistratlon  variable  to  be  axaained  both  between  and  within  subjects  for  all 
six  subtests,  thus  allowing  greater  flexibility  of  analysis. 


PC  eubteet.  Tha  data  collection  plan  for  the  PC  subtest  is  shown  in  Fig¬ 
ure  1.  To  overcome  tha  problem  of  lengthy  PC  items,  a  scrolling  procedure  was 
devised  that  enabled  examinees  to  move  forward  and  backward  through  the  item 
text  a  line  at  a  time.  Scrolling  was  activated  by  depressing  a  designated 
forward  or  backward  scrolling  key  which  erased  the  text  froa  the  current 
screen  (field)  and  replaced  it  with  a  different  screen  (field)  of  text.  De¬ 
pending  on  the  scrolling  key  chosen  (forward  or  backward),  exaainees  were  able 
to  view  new  or  previously  presented  material. 


Three  coepute r-adninl s trat ion  Mode  conditions  (CRT-1  to  CRT-3)  were  in¬ 
cluded  to  essess  the  effectiveness  of  three  possible  solutions  to  this  complex 
item  presentation  problem.  In  the  first  Mode  condition  (CRT-1)  the  paragraph 
under  consideration  could  be  scrolled  on  the  screen  while  one  question  at  a 
time  appeared  beneath  it  in  a  separate,  nonscrollable  field.  Previous  ques¬ 
tions  about  the  paragraph  ware  erased  following  an  examinee's  response  and 
before  the  next  question  appeared,  and  examinees  were  unable  to  refer  to  pre¬ 
vious  paragraphs  or  previous  questions  pertaining  to  the  current  paragraph. 

The  second  Mode  condition  (CRT-2)  presented  the  paragraph  and  all  ques¬ 
tions  related  to  it  in  a  single  scrollable  field  with  no  access  to  previous 
paragraphs  or  their  questions.  This  condition  allowed  the  examinee  to  use  the 
entire  screen  to  view  the  paragraph  or  single  questions,  and  to  move  back  and 
forth  between  the  current  paragraph  and  its  questions.  The  third  Mode  condi¬ 
tion  (CRT-3)  allowed  the  current  paragraph  to  be  scrolled  on  the  screen,  and 
beneath  it  questions  could  be  scrolled  separately.  The  displayed  paragraph 
would  change  automatically  as  the  items  ware  scrolled  to  remain  current  with 
the  displayed  item,  so  that  an  examinee  could  scroll  beck  to  any  previous 
paragraph  and  its  questions.  This  condition  also  provided  an  answer-sheet 
type  of  display  at  tha  top  of  the  screen  which  allowed  examinees  to  monitor 
their  progress  through  the  test,  and  permitted  them  to  go  back  and  change  an¬ 
swers  to  all  previous  PC  items  at  any  tins  throughout  the  test. 

Bach  presentation  condition  appeared  twice  in  the  first  two  experimental 
groups,  once  with  Version  A  and  alternatively  with  Version  B,  for  a  total  of 
six  conditions  with  50  to  60  exaainees  per  condition.  Due  to  the  varied  ex¬ 
perimental  conditions  and  limitations  in  sample  size,  it  was  necessary  to  lim¬ 
it  the  administration  of  the  PC  subtest  to  Form  11  of  the  ASVAB  in  Groups  1 
and  2.  Group  3,  the  paper-and-pencil  only  group,  was  not  restricted  by  the 
three  computer  conditions;  so  the  three  forms  of  the  PC  subtest.  Versions  A 
and  B,  were  administered  Instead,  for  a  total  of  six  conditions  with  50  to  60 
exaainees  per  condition. 
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MO  >ad  CS  wbttiti.  Figure  2  describee  the  date  collection  plan  for  the 
MO  end  CS  subtest*.  Here,  two  computer  presentation  Mode  conditions  were  In¬ 
cluded  (CRT-1  end  CRT-2)  which,  when  combined  with  Versions  A  end  B  of  each 
subtest,  provided  four  conditions  per  group  with  80  to  85  examinees  per  condi¬ 
tion,  In  the  CRT-1  condition,  the  lteas  appeared  on  the  CRT  one  at  a  tine, 
end  the  exaalnees  were  Instructed  to  answer  as  aany  of  the  lteas  as  possible 
within  the  allotted  tine.  In  the  CRT-2  condition,  several  lteas  were  dis¬ 
played  at  a  tins  on  the  screen  with  the  sane  Instructions;  for  MO,  three  lteas 
appeared  on  the  screen  at  the  sane  tiae,  and  for  CS,  two  blocks  of  seven  lteas 
were  displayed.  Response  tlass  were  recorded  at  both  the  single-  and  multi¬ 
ple-!  tea  level.  Again,  limitations  In  sample  size  restricted  the  edalnlstra- 
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tlon  of  tho  HO  and  CS  subtost s  to  Pom  12  In  Groups  1  and  2.  Group  3  received 
both  versions  of  all  throa  forms  for  a  total  of  six  conditions  with  50  to  60 
examinees  par  condition. 

AS.  MC.  and  P  aubtasta.  Tha  data  col lac tlon  plan  for  tha  AS,  MC,  and  El 
subtests  is  shown  in  Flgura  3.  Unlike  tha  othar  aubtasta,  thasa  consistad  of 
only  a  single  conputar  prasantatlon  condition  that  praaantad  all  ltans  includ¬ 
ing  graphical  images.  This  paruittad  tha  administration  of  Forma  11,  12,  and 
13  to  all  thraa  groups,  for  a  total  of  six  conditions  par  group  with  50  to  60 
examinees  par  condition.  Tha  graphical  laagas  for  tha  computar-adainiatarad 
ASVAB  tasts  wars  developed  by  tha  four-step  procaas  of  (a)  digitising,  (b) 
aditlng,  (c)  mowing,  and  (d)  concatanatlon  and  lndaxing.  Tha  digitising  pro- 
cass  was  done  using  a  Kurts  Sarias  Two  12*  x  17*  digitiser  pad. 

Tha  dlgltlsar  program  that  was  written  for  this  project  (called  DIGITI¬ 
ZER)  allowed  for  digitising  to  a  half  screen  while  scaling  tha  image  to  make 
it  as  large  as  possible.  In  this  way,  tha  limited  120  vertical  dots  by  320 
horlsontal  dots  half-screen  graphics  window  was  used  as  completely  as  possi¬ 
ble.  DIGITIZER  allows  points,  lines,  and  connected  lines  ("connect  the  dots" 
or  polygon  node)  to  be  plotted,  either  entered  individually  or  "streamed"  to¬ 
gether. 

After  digitising,  all  images  were  cleaned  up  and  completed  using  tha 
graphics  editor.  Tha  graphics  editor  used  was  a  reworked  version  of  the  Terak 
graphics  editor  GREDIT.  It  allows  lines,  points,  circles,  arcs,  and  arrow 
heads  to  be  drawn  or  erased,  or  text  to  be  entered  or  edited.  It  also  allows 
the  superimposing  of  one  image  upon  another.  Ultimately,  all  images  were 
superimposed  in  this  fashion,  with  one  image  on  the  top  of  the  screen  and  the 
other  on  the  bottom. 

Once  the  image  was  refined  to  the  desired  level,  it  was  centered  within  a 
half-screen  area  and  located  in  the  proper  half  screen  (top  or  bottom),  using 
a  program  called  HOVER,  which  also  allowed  the  duplication  of  one  part  of  the 
screen  on  another  part  of  the  screen.  The  graphical  portions  of  the  screens 
were  then  concatenated  and  indexed  into  a  random  access  file  that  was  utilised 
by  the  test  administration  program  to  retrieve  graphical  images  associated 
with  text  segments. 


Combined  Conditions 


Data  Analysis 


The  experimental  design  plan  provided  for  three  test  administration 
groups:  Group  1,  to  be  tested  by  computer  followed  by  paper-and-pendl;  Group 
2,  tested  by  paper-and-pencil  followed  by  computer;  and  Group  3,  tested  by 
paper-end-pencil  twice.  Within  each  group  the  three  forms  and  two  versions  of 
each  subtest  were  administered,  except  for  PC  in  the  first  two  groups  where 
the  three  screen  Modes  (CRT-1  to  CRT-3)  were  substituted  for  three  forms  and 
all  examinees  received  Form  11.  This  resulted  in  18  experimental  conditions 
(3  groups  x  3  forms  x  2  versions).  In  addition,  for  the  conputer  administra¬ 
tion  in  Groups  1  and  2,  two  separata  screen  conditions  wars  administered  for 
the  HO  and  CS  subtests,  thus  adding  an  additional  12  conditions,  yielding  a 
total  of  30  experimental  conditions. 
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The  model  ehoitn  foe  the  AN  OVA  mi  a  repeated  aaaauraa  multivariate  (KA¬ 
MOVA)  das i(n  aalnf  total  subtest  seoraa  as  dapandant  variables,  with  uni¬ 
variate  follow-up  ANOVAs  for  all  sl(nlflcant  affacts.  A  multivariate  analysis 
waa  aalactad  bacausa  It  offars  a  subataatlal  advantage  with  raspaet  to  power, 
whlla  controlling  tha  Typa  1  arror  rata  over  all  subtasts  slnoltanaoualy  at 
tha  noalnal  alpha  level  (Morrison,  1976). 


▲  fundamental  assumption  undo r lying  any  MAN OVA  analysis  is  that  of  homo- 
ganalty  of  tha  covariance  aatrlcas  within  aach  call.  Howavar,  tha  KAMOVA  nod¬ 
al  la  wldaly  racognisad  to  ba  robust  to  violation  of  this  assumption,  espe- 
dally  for  large  saaple  slaas.  Furtharaora,  anltlrarlata  tests  of  honoganaity 
are  vary  powerful,  given  large  saaplas,  often  leading  to  a  rejection  of  tha 
hypothesis  of  equal  covarlanca  aatrlcas  on  tha  basis  of  ainor  differences 
(Cooley  &  Lohnes,  1971).  Univariate  AMOVA  models  In  a  similar  way  assuae  ho¬ 
mogeneity  of  variances,  and  in  a  Ilka  Banner,  are  largely  robust  to  violations 
of  this  assumption.  This  situation  ho Ida  at  least  for  be  tween-subjects  facto¬ 
rial  aodela,  but  for  repeated  measures  designs  violations  of  tha  equal  vari¬ 
ance  assumption  have  bean  shown  to  load  to  a  greatly  Increased  probability  of 
type  I  error  (O'Brien  A  Kaiser,  1985).  Unfortunately,  there  la  no  known  test 
at  present  for  the  homogeneity  of  repeated  measures  variances.  Therefore,  the 
best  defense  against  an  Increased  Incidence  of  Type  I  error  Is  to  Interpret 
significant  outcomes  accordingly,  when  the  variances  over  sessions  appear  to 
be  heterogeneous. 


KAMOVA  is  based  upon  a  comparison  of  the  latent  structures  of  the  be- 
tween-groups  sums  of  squares  and  cross-products  (SSCP)  matrix,  H,  and  the 
wlthln-groups  SSCP  matrix  of  subtest  scores,  E,  for  a  given  effect.  This  Is 
accomplished  through  the  decomposition  of  the 'product  matrix,  HE-1,  and  the 
analysis  of  the  resulting  latent  roots.  Although  there  Is  no  uniformly  most 
powerful  statistic  for  conducting  this  analysis,  all  of  the  widely  accepted 
statistics  are  aoma  function  of  the  latent  roots  of  g?'1,  and  tha  choice  of  an 
appropriate  statistic  la  dependent  upon  the  rank  of  the  matrix,  or  the  number 
of  significant  roots  obtained. 


Vor  the  situation  where  the  rank  of  the  matrix  is  1,  the  most  powerful 
test  of  significance  has  been  shown  to  be  Roy's  Largest  Root  test;  whereas, 
when  the  rank  Is  greater  than  1,  the  Plllal-Bartlett  Trace  V  test  Is  most  pow¬ 
erful  (Olson,  1976).  The  PUlai-Bartlett  offers  the  additional  advantage  of 
being  the  most  robust  to  the  violation  of  the  assumptions  of  the  KAMOVA  model, 
whereas  Roy's  test  is  extremely  sensitive  to  violation  of  the  homogeneity  as¬ 
sumption. 


for  the  present  analyses,  four  multivariate  tests  of  significance  were 
applledi  Roy's  Largest  Root,  Hotelling's  T,  Wilks'  Lambda,  and  tha  Plllal- 
Bartlett  Trace  V  test,  for  every  experimental  outcome,  the  results  of  all 
four  tests  were  In  complete  agreement.  Therefore,  the  Plllal-Bartlett  Trace  V 
statistic,  which  has  been  recommended  for  general  use  (Olson,  1976)  Is  report¬ 
ed  for  the  KAMOVA  analyses. 


The  Plllal-Bartlett  Trace  V  test  Is 

V  -  ?C1/ (1+C1) 


(1) 


where  C.  is  the  ith  latent  root  of  tha  matrix  H(  *  and  S  ia  lta  rank.  Plllai 
derived  an  approximation  to  tha  F  distribution  (Morrison,  1976)  for  V  as 


„  (dfe  -  p  +  S)V 
b(S  -  V) 


(2) 


with  Sb  and  S(dfa  -  p  +  s)  degrees  of  freedoa,  where  dfe  is  the  degrees  of 
freedoa  within  groups,  p  is  the  number  of  dependent  variables,  S  is  the  rank 
end  b  is  the  larger  of  p  and  the  degreea  of  freedoa  between  groups. 


of  wr1, 


The  follow-up  analysis  for  those  KAMOVA  outcomes  found  to  be  significant 
was  based  on  univariate  ANOYA  F  tests  to  identify  the  specific  subtests  for 
which  the  effects  were  significant.  Once  a  MAMOVA  result  is  found  signifi¬ 
cant,  however,  all  follow-up  statistical  tests  are  unconstrained  in  terns  of 
Type  I  error.  That  is,  the  Type  I  error  rate  applies  at  the  individual  test 
level,  and  over  nuaerous  tests,  the  rate  is  compounded.  Caution  oust  be  ex¬ 
ercised  in  such  a  case  in  interpreting  significant  outcomes.  Furthermore, 
standard  foras  of  post  hoc  statistical  tests,  while  varying  in  power  and  flex¬ 
ibility,  are  all  Increasingly  sensitive  with  increasing  saaple  sixes,  and  for 
the  present  study,  relatively  large  saaples  were  required  to  adequately  pro¬ 
vide  for  the  correlational  analyses,  such  as  factor  analysis.  Such  large  saa¬ 
ples  serve  to  Increase  the  sensitivity  to  post  hoc  statistical  tests  to  the 
point  where  meaningless  differences  become  prominent  and  obscure  the  Important 
group  differences  under  experimental  manipulation. 


Paper-mi-peacil  baseline  analysis.  The  purpose  of  the  first  analysis 
was  to  examine  the  effaces  across  all  six  subtests  of  (a)  Form— the  three  test 
forms — ASVAB  Forms  11,  12,  and  13;  (b)  Version— the  two  Versions  (A  and  B) 
within  each  form;  and  (b)  Session — the  repeated  measurement  of  paper-and- 
pencll  tests.  This  analysis  was  confined  to  Group  3  (N  »  333),  which  func¬ 
tioned  as  a  baseline  comparison  group,  being  tested  entirely  in  the  paper-and- 
pencil  medium.  Figure  A  shows  the  experimental  design  plan  for  Group  3. 


The  analysis  consisted  of  a  repeated  measures  MANOVA  with  univariate  ANO- 
VA  follow-up  tests  for  all  significant  effects  across  session  within  subjects, 
and  test  fora  and  test  version  between  subjects.  The  dependent  variables  were 
the  total  scores  for  each  of  the  six  subtests  taken  by  each  exaalnee  (PC,  NO, 
CS,  AS,  MC,  and  El). 


Paragraph  Comprehension.  The  second  analysis  was  designed  to  determine 
if  differences  were  observed  in  aean  test  scores  among  the  three  modes  of  com¬ 
puter  presentation  of  the  PC  subtest  between  subjects,  and  between  the  comput¬ 
er  and  paper-and-pencil  presentation  media  both  between  and  within  subjects. 
Test  form  differences  were  also  examined  to  eliminate  confounding  with  the 
other  effects. 


Figure  3  shows  the  experimental  design  plan  for  the  PC  analysis.  All 
examinees  were  tested  using  Fora  11  of  the  PC  subtest  to  allow  sufficient  sam¬ 
ple  size  for  a  fully  crossed  design.  Group  1,  consisting  of  332  examinees, 
was  subdivided  into  three  groups  of  approximately  equal  size.  Each  subgroup 
was  administered  one  test  version  in  one  mode  of  the  computer  medium  in  the 
first  testing  session,  followed  by  paper-and-pencil  testing  with  the  alternate 
version  in  the  second  session;  subgroups  of  between  50  and  60  examinees  within 
each  node  subgroup  were  administered  Versions  A  and  B  of  ASVAB  Fora  11  PC  in 
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Figure  4.  Experimental  Design  Plan  for  Paper-and-Pencil  Baseline  Analysis. 


counterbalanced  order.  Group  2  (N  ■  332)  was  tested  with  the  presentation 
aadla  In  reverse  order,  l.e.,  pape r-and-penc 11  followed  by  coaputer  presenta¬ 
tion. 


The  net hod  of  analysis  was  a  repeated  measures  MANOVA  with  the  PC  subtest 
score  as  the  sole  dependent  variable.  The  effects  that  were  examined  as  be- 
tween-subjects  variables  for  the  PC  analysis  included  three  modes  of  computer 
presentation  (Mode),  coaputer  versus  pape r-and-penc 11  presentation  (Medium), 
and  test  version  (Version).  The  affect  examined  as  a  wi thin-subject s  variable 
was  the  coaputer  versus  paper-and-pencll  presentation  medlua  across  sessions 
(Session). 


Speeded  tests.  The  purpose  of  the  third  analysis  was  to  determine  If 
differences  existed  between  asan  scores  resulting  from  the  two  modes  of  coa¬ 
puter  presentation  of  the  MO  and  CS  subtests,  and  between  the  computer  and 
paper-and-pencll  presentation  media  both  between  and  within  subjects.  Test 
version  differences  were  examined  to  determine  If  they  interacted  with  these 
primary  effects. 


Figure  6  summarizes  the  experimental  design  plan  for  these  analyses.  All 
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Figure  5.  Experimental  Design  Plan  for  Analysis  of 
Paragraph  Comprehension  Subtest. 
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Figure  6.  Experimental  Design  Plan  for  Analysis  of  Speeded  Subtests. 


testing  was  limited  to  Form  12  to  allow  sufficient  sample  size  for  a  fully 
crossed  design.  Group  1,  with  332  examinees ,  was  subdivided  randomly  into  two 
subgroups  of  approximately  equal  size.  Each  subgroup  was  administered  one 
test  in  one  of  the  two  computer  Nodes  (single-item  or  multiple-item  screen)  in 
the  first  testing  session,  followed  by  paper-and-pendl  testing  with  the  al¬ 
ternate  version  of  Form  12  in  the  second  session.  Group  2,  also  with  332  ex¬ 
aminees,  was  tested  in  the  reverse  order,  with  paper-and-pencil  testing  in  the 
first  session  and  computer  testing  in  the  second  session.  Within  each  of  the 
Mode  subgroups  of  Group  1  and  Group  2,  half  of  the  examinees  were  assigned  to 
either  Version  A  or  Version  B  of  ASVAB  Form  12  in  random  order  to  test  the 
Version  effect.  This  yielded  a  total  of  eight  experimental  groups  of  80  to  85 
examinees  each.  (Due  to  sample  size  limitations,  within-subject  test  version 
and  medium  effects  were  necessarily  combined  into  a  single  experimental  condl- 


The  method  of  analysis  was  a  rapaatad  aaasuras  HAM OVA  with  tha  MO  and  CS 
subtast  scoras  as  tha  dependent  variables,  with  univariate  ANOVA  follow-up 
tasts  for  all  significant  tests.  Tha  betweenrsubjects  variablas  wore  test 
▼anion  (Version),  coaputer  presentation  node  (Node),  and  coaputer  versus  pa- 
par-and-pancil  presentation  aediua  (Mediua),  and  tha  within-subjacts  variable 
was  tha  coaputer  vsnus  papar-and-pancil  nediua  (Session)  affect. 

Graphical  subtests.  Tha  fourth  analysis  was  designed  to  exaaine  tha  dif¬ 
ferences  between  tha  coaputer  and  papar-and-pancil  presentation  aadia  for  tha 
AS,  MC,  and  El  subtasts.  These  differences  wan  analysed  for  Forms  11,  12, 
and  13  of  each  subtast  and  by  van  ion.  Medium  and  Version  affects  wan  axaa- 
ined  in  a  completely  unconfoundad  design.  For  tha  graphical  subtasts,  a  par¬ 
ticular  question  of  intenst  is  what  differences  in  performance,  if  any,  re¬ 
sult  froa  tha  presentation  of  graphical  laages  on  a  coaputer  screen  versus  the 
printed  page.  The  analysis  of  the  MC  subtest  addresses  this  question  direct¬ 
ly,  since  of  the  25  lteas  comprising  the  test,  23  for  Fora  11,  21  for  Fora  12, 
and  22  for  Fora  13  contain  graphical  laages.  In  addition,  tha  MC  subtasts,  as 
with  all  others,  were  randoaly  distributed  over  28  coaputen  to  control  for 
individual  screen  resolution  differences. 

Figure  7  shows  the  experimental  design  plan  for  these  tests.  The  332 
examinees  in  Group  1  were  randoaly  assigned  to  three  subgroups.  Each  subgroup 
was  administered  one  fora  of  each  subtest,  one  version  in  tha  computer  medium 
in  Session  1  and  the  alternate  version  in  the  paper-and-pencll  aediua  in  Ses¬ 
sion  2.  The  second  group  of  332  examinees  was  tested  in  a  similar  manner  but 
with  the  order  of  presentation  medium  reversed. 

The  analysis  was  a  repeated  measures  MAMOVA  with  total  subtest  scores  as 
dependent  variables.  The  effects  that  were  exaained  as  between-subjects  vari¬ 
ables  Included  test  form,  computer  versus  paper-end-pencil  presentation  madl¬ 
ua,  and  test  version.  The  effect  exaained  as  a  within-subjacts  variable  was 
the  computer  versus  paper-and-pencll  presentation  medium. 

Reliability  Analyses 

Experimental  group  and  session  abbreviations  are  presented  in  Figure  8. 
The  first  character  stands  for  aediua  of  administration:  C  for  coaputer,  P  for 
paper-and-pencil  adninlstration;  the  second  character  denotes  the  experlaental 
group  (1,  2,  or  3);  the  third  digit  indicates  testing  session  1  or  2.  Due  to 
the  nature  of  the  data  collected,  two  types  of  reliability  coaparisons  ware 
aade. 


Test-retest.  The  first  comparison  used  a  test-retest  correlation  design 
and  was  computed  for  all  the  ASVAB  subtests.  The  subtests  CS,  AS,  MC,  and  El 
contain  the  same  items  in  both  Version  A  and  Version  B,  providing  true  test- 
retest  data.  However,  subtests  PC  and  MO  use  different  lteas  in  Versions  A 
and  B,  thus  changing  the  comparison  slightly  to  one  of  an  alternate-forms  cor¬ 
relation,  although  tha  versions  are  noainally  parallel.  The  comparison  of 
interest  was  that  between  Groups  1  and  2  (Cll  then  P12,  and  P21  then  C22)  ver¬ 
sus  the  peper-and-pencil  only  (P31  then  P32)  group.  In  these  coaparisons  the 
Group  3  reliabilities  served  as  a  baseline  against  which  the  other  experlaen¬ 
tal  conditions  could  be  judged.  The  reliabilities  were  compared  by  t-tests 
for  each  contrast  after  performing  Fisher's  (1921)  r  to  z  transformation  on 
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Figure  7.  Experimental  Design  Plan  for  Analysis  of  Graphical  Subtests. 
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Test  Administration 


Figure  8.  Overview  of  Tsst  Conditions  by  Group  snd  Session 
for  Computer  (CRT)  snd  Paper-and-Pencll  (P&P)  Groups. 


tbs  test-retest  correlations. 

Internal  consistency.  The  final  reliability  comparisons  were  made  using 
an  Internal  consistency  measure  of  reliability,  Cronbach's  (1951)  coefficient 
alpha,  which  reduced  to  the  Kuder»Rlchardsou  (1937)  Formula  20  (KR-20)  due  to 
the  dichotomous  item  responses.  Because  Internal  consistency  reliability  es¬ 
timates  are  not  appropriate  for  speeded  tests,  only  subtests  PC,  AS,  MC,  and 
El  were  considered  In  these  comparisons. 

The  data  collection  plan  provided  the  means  to  perform  both  matched 
groups  (within-group  across  sessions)  and  multiple  independent  groups  (acroes- 
g roups  within  session)  KR-20  comparisons.  In  order  to  perform  the  complete 
matched  group  comparison  across  sll  subtests  by  Form  end  Version  for  each  ex¬ 
perimental  group,  72  tests  of  the  type  proposed  by  Feldt  (1980)  would  be  nec¬ 
essary.  A  simple  independent  groups  comparison  for  each  subtest  by  form  and 
version  within  each  session  would  require  that  48  tests  using  Hakstlan  and 
Whalen's  (1976)  method  be  made.  The  sheer  number  of  statistical  tests  neces¬ 
sary  to  adequately  compare  the  KR-20  reliabilities  suggested  that  the  findings 
obtained  would  be  extremely  difficult  to  Interpret  snd  confounded  by  Type  1 
error. 

As  an  alternative,  the  KR-20  estimates  were  averaged  separately  for  each 
subtest  within  each  group  and  session  (Cll,  P12,  P21,  C22,  P31,  and  P32)  and 
examined  for  differences.  Collapsing  the  data  In  this  way  was  justified  be¬ 
cause  the  ASVAB  forms  and  versions  are  designed  to  yield  similar  msasuremsnt 
characteristics}  thus,  any  Important  reliability  differences  across  presenta¬ 
tion  medium  or  session  would  be  consistent  across  all  versions  and  forms. 

Studying  the  reliabilities  in  this  manner  allowed  the  discovery  of  possi¬ 
ble  trends  or  differences  In  KR-20s  both  within  groups  across  sessions,  and 
across  groups  within  sessions.  As  In  the  test-retest  analysis,  the  paper-snd- 
pencll-only  groups  (Group  3,  P31  and  P32)  were  used  as  a  baseline  against 
which  the  mean  KR-20*  from  the  other  groups  were  judged. 
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tests  at  the  subtest  score  level,  under  both  edelnls tret Ion  nedlua  conditions* 
The  data  were  correlation  and  covariance  aetrlces  for  betueen-subtest  scores 
coaputed  froa  exealnees'  total  scores  for  each  of  the  six  subtests. 

One  aethod  used  to  deteralne  the  equivalence  of  test  presentation  asdla 
was  the  axaalnatlon  of  the  factor  structure  of  the  betueen-subtest  correla¬ 
tions  across  conditions*  For  this  coaparlson,  unrotated  principal  factor 
analysis  was  used  as  the  factoring  as t hod.  Although  sets  of  coaaon  factor 
loadings  ere  usually  not  unique  and  natty  different  sets  of  loading  values  can 
define  a  solution,  principal  factor  analysis  yields  a  factor  solution  that 
defines  both  a  unique  coaaon  factor  space  and  a  unique  set  of  factor  loadings 
(Haraan,  1976).  Therefore,  factor  loadings  froa  the  unrotated  principal  fac¬ 
tor  analysis  solution  (definad  by  extraction  of  asxlaua  variance  froa  the  cor¬ 
relation  aetrlces  with  squared  Multiple  correlations  on  the  diagonals)  of  the 
subtests  were  directly  coapared  for  equivalence  across  media  of  presentation. 

The  equality  of  betueen-subtest  covariance  aetrlces  end  factor  structures 
was  exaalned  using  the  asxlaua  likelihood  Methods  of  LISREL  VI  (Joreskog  & 
Sorboa,  1984).  There  ere  three  aaln  indices  of  general  aodel  fit  yielded  by 
LISREL.  The  first  is  the  overall  chi-square  (Xz).  The  test  aade  by  chi- 
square  judges  the  fit  of  the  constrained  aodel  against  the  alternative  hypoth¬ 
esis  that  the  estimated  covariance  Matrix  is  unconstrained.  Degrees  of  free- 
doa  for  X2  are  calculated  by 

df  -  %k(k  +  1)  -  t  (3) 

where  k  is  the  nuaber  of  obeerved  variables,  and  t  is  the  nuaber  of  free  pa- 
raasters  estimated.  Joreskog  and  Sorboa  (1984)  suggest  that  X2  be  used  as  an 
index  of  the  degree  of  aodel  fit  and  not  strictly  as  a  test  statistic. 

The  second  index  of  overall  aodel  fit  is  the  goodness-of-flt  Index  (GFI). 
GFI  is  defined  by 

tr(£”1S  -  I)2 

GFI  "  1 - -  <4> 

tr(I-1§) 

where  tr  is  the  trace  of  the  indicated  aetrlces,  £  is  the  fitted  aatrlx,  §  Is 
the  obeerved  covariance  aatrlx,  and  X  is  an  Identity  aatrlx.  The  range  of  GFI 
is  between  aero  and  1.0,  and  It  is  a  Measure  of  the  aaount  of  covariance  and 
variance  accounted  for  by  the  aodel. 


as 


The  last  aodel  fit  Index  is  the  root  aean  square  residual  (RMR)  defined 


where  k  Is  tbs  number  of  obssrved  variables  in  tbs  model,  Sjj  is  tbs  obosrvsd 
variance  or  covariance,  and  a**  is  tbs  as t Isa tad  covariance  or  varlanea  coa- 
poaant.  Tbs  Into rp rotation  or  RMR  dapaads  on  tbs  ralatlva  slsas  of  tbs  covar- 
laaeaa  and  variances  of  tba  obsarvad  varlablas.  for  example,  a  largo  valua 
for  BMk  as  eoaparad  to  tba  avarsga  obsarvad  eovarianca  or  varlaaca  would  ba  an 
indication  of  a  nodal  that  did  not  fit  tba  data  vary  wall. 

Joraskog  (1971)  outlined  a  not bod  by  which  tba  factor  structures  of  tost 
batteries  could  ba  tasted  for  equality  across  different  groups  of  examinees. 
Models  assuming  different  levels  of  equalities  ware  tasted  sequentially  until 
tba  level  of  structured  equality  appropriate  for  tba  betweet^subtest  data  was 
found.  The  first  modal  tasted,  tba  most  strict  test  of  covariance  equality, 
assumed  that,  within  each  Session  (1  and  2),  the  be tween-sub test  scores  covar¬ 
iance  matrices  for  the  paper-and-penc 11  administrations  ware  equal  for  each 
group  (Session  1:  ZG2  ■  z^3  and  Session  2:  Eqj  ■  Eq3).  This  test  of  tba  nodal 
that  the  covariance  structures  within  the  paper-enc-pencil  administrations 
ware  equal  across  groups  provided  a  baseline  against  which  the  covariance 
structures  of  coaputer-ad ministered  tests  could  ba  judged.  Next,  the  same 
test  was  made,  again  by  session,  with  the  addition  of  tba  covariances  for  tba 
computer  administrations  (?G1  -  ZG2  -  ZG3).  These  tests  are  generalizations 
of  tbs  Bartlett  test  for  homogeneity  of  variances  (Morrison,  1976)  end  are. 
susceptible  to  having  a  high  degree  of  power  when  sample  alias  are  large, 
causing  rejection  of  the  null  hypothesis  when  minor  differences  are  present 
(Cooley  &  Lohnes,  1971). 

The  next  model  tested,  holding  a  much  less  strong  equality,  was  that 
within  each  Session  (1  and  2)  the  be tween-sub test  scores  yielded  the  same  fac¬ 
tor  structure  in  each  group. 

Mlthltt-aubtaat  structure.  The  analysis  of  the  similarity  of  factor 
structures  within  subtests  was  also  performed  for  subtests  AS,  MC,  end  El. 
These  were  the  only  non-spa eded  subtests  containing  the  same  items  in  both 
versions  (A  and  B).  By  combining  examinees  tested  on  either  version,  the  sam¬ 
ple  size  requirements  of  factor  analysis  were  nearly  met,  with  approximately 
100  persons  per  subtest  available  across  Forms  (11,  12,  and  13)  within  each 
cell  (Cll,  P12,  P21,  C22,  P31,  P32)  of  the  experimental  design. 

The  factor  structures  were  compared  through  uniterated  principal  factor 
analysis,  as  described  previously,  of  the  item  Intercorrelation  matrices  com¬ 
posed  of  phi  correlations.  Five  unrotated  principal  factors  were  extracted 
from  each  correlation  matrix.  These  comparisons  were  made  across  both  sub¬ 
jects  end  media  using  the  first  session  baseline  paper-end-pencil  group  for 
all  comparisons  (Cll  and  C22  versus  P31).  The  subtest  factor  structures  from 
both  computer-administered  conditions  were  collared  against  the  subtest  factor 
structures  of  the  Session  1  paper-and-pencil-only  group  (Group  3)  because  it 
yielded  an  adequate  representation  of  the  factor  structures  of  the  subtests  in 
ell  the  paper-and-pencil  groups. 


Item  Analysis 


Due  to  limitations  in  sample  size  (between  50  and  75  per  cell),  only 
classical  test  theory  item  parameters  (polnt-blserlal,  biserial,  and  propor¬ 
tion  correct)  were  calculated  end  analyzed  for  the  non-speeded  subtests.  The 
sample  size  demands  of  item  response  theory  item  parameterization  using  LOGIST 
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(Wood,  Wings rsky,  &  Lord,  1976) — 500  to  1,000  per  coll  (Wood  &  Lord,  1976)— 
greatly  exceed  tho  number  of  examinees  in  this  study. 

Ths  subtosts  undsr  study  were  chossn  bscsuss  thsy  srs  tbs  most  problema- 
tle  ASVAB  subtosts  for  eooputsr  sdulnl strut ion.  Of  psrtlculsr  coaeorn  wss  tho 
possiblo  dlfforonco  In  noosursaont  proportlos  of  items  containing  graphical 
loogos  (in  subtosts  AS,  MC,  and  El)  duo  to  problons  In  translotlng  tho  lnogo 
f roo  papor  to  a  cooputor  CRT  scroon.  This  question  of  aaasureaent  equivalence 
was  noatly  addrossod  at  tho  subtost  level  bocauso  of  tho  distribution  of 
graphical  content  itoas  anong  tho  throe  tests.  Tho  HC  subtost  contains  bo* 
tueen  21  to  23  graphical  itoas  (23  in  Fora  11,  21  in  Fora  12,  22  in  Fora  13) 
out  of  tho  25  total,  uhoroos  subtost  El  has  only  two  or  throe  out  of  20  itoas. 
There fora,  any  coaputor  adainlstratlon  affect  on  graphical  itoas  would  iapect 
on  subtost  HC  in  its  entirety  and  causa  aoan  differences  across  asdla,  which 
would  bo  found  through  tho  MAN0VA  analysis.  Subtost  El,  hawing  alaost  no 
graphical  content  itoas,  was  used  to  index  possible  graphical  ltoa  adainlstra¬ 
tlon  differences  since  it  has  slailar  tost  objectives,  being  e  non-speoded, 
technical  information  tost.  Subtost  AS  was  not  used  in  this  analysis  bocauso 
it  contains  several  graphical  itoas  (5  in  Fora  13,  and  6  in  Foras  11  and  12), 
disqualifying  it  as  either  a  high  or  low  graphical  content  subtest. 

Adainlstratlon  asdlua  differences  for  graphical  content  itoas  were  also 
compared  at  the  factor  structure  level.  The  differences  in  the  unrotated 
principal  factor  solutions  for  subtest  MC  across  media  of  presentation  were 
compared  with  the  factor  structure  differences  for  subtest  El  across  the  same 
conditions.  Any  large  discrepancy  between  MC  and  El  factor  structure  differ¬ 
ences  would  provide  evidence  that  coaputor  administration  changes  the  interre¬ 
lationships  aaong  graphical  items.  Implying  that  the  translation  of  laages 
from  paper  to  CRT  screen  differentially  affects  itsas  with  graphical  content. 


Table  2  shows  the  outcome  of  the  MANOVA  for  the  paper-and-pendl  baseline 
group,  both  within  and  across  sessions,  for  all  subtests.  Significance  (p  < 
•05)  is  indicated  for  one  within-sesslon  factor,  test  fora  (Fora),  the  repeat¬ 
ed  measures  factor  (Session),  and  for  the  interaction  of  test  version  (Ver¬ 
sion)  with  Fora  and  Session.  All  other  factors  and  interactions  were  not  sig¬ 
nificant;  therefore,  no  further  analyses  were  necessary. 

The  results  of  the  univariate  follow-up  analyses  by  subtest  for  the  sig¬ 
nificant  effects  identified  in  the  nultlvarlate  baseline  analysis  are  given  in 
Table  3.  For  the  Fora  factor,  algniflcanca  was  found  for  the  AS  subteat. 
Significant  Session  effects  are  shown  for  the  NO,  CS,  snd  El  subtests,  and  PC 
shows  a  significant  outcome  for  the  Version  by  Fora  by  Session  interaction. 

The  next  step  in  the  analysis  was  to  exaalne  the  difference  in  the  asans 
for  each  level  of  each  effect.  Tables  4  through  9  show  the  asans  for  all  sub¬ 
tests  by  condition.  A  significant  difference  in  the  asan  scores  by  Fora  was 
found  for  the  AS  subtest;  the  largest  of  these  was  between  Fora  11  and  Fora 
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Table  2.  Pillai-Bartlett  Trace  Values,  Degrees  of  Free doe, 
Approximate  F  Ratios,  and  Estimated  Significance  Levels  (p) 
for  the  Baseline  MANOVA  (N  -  333) 


Fji  wnarRX 


I 


Source  of  variation 

Trace  V 

Degrees  of 
Between 

freedom 

Within 

F 

P< 

Between  Groups 

Font 

.10 

12 

646 

2.96 

.001** 

Version 

.01 

6 

322 

.30 

.935 

Font  x  Version 

.04 

12 

646 

1.11 

.353 

Within  Groups 

Session 

.35 

6 

322 

29.21 

.001** 

Fora  x  Session 

.05 

12 

646 

1.45 

.138 

Version  x  Session 

.02 

6 

322 

.93 

.473 

Version  x  Form  x  Session 

.08 

12 

646 

2.30 

.007** 

s’ 

V 

jj 

5 


♦♦Statistically  significant  at  p  <  .01. 


Table  3.  Results  of  Univariate  ANOYAs  for  Significant  Effects 
Identified  in  the  Besellne  MANOVA  (N  -  333) 


Effect,  degrees 
of  freedom, 
and  subtest 

Mean 

Between 

squares 

Within 

F 

P< 

Form  (2,327) 

PC 

6.82 

8.96 

.76 

.468 

NO 

160.97 

97.17 

1.66 

.192 

CS 

317.30 

303.33 

1.05 

.353 

AS 

129.51 

39.02 

3.32 

.037* 

MC 

10.74 

33.05 

.33 

.723 

El 

20.22 

23.13 

.87 

.418 

Session  (1,327) 

PC 

.29 

3.01 

.10 

.755 

NO 

312.22 

14.92 

20.93 

.001** 

CS 

5506.34 

34.10 

161.47 

.001** 

AS 

.38 

1.18 

.33 

.568 

MC 

5.23 

2.07 

2.53 

.113 

El 

12.99 

1.85 

7.02 

.008** 

Version  x  Form  x  Session  (2,327) 

PC 

32.99 

3.01 

10.94 

.001** 

NO 

26.34 

14.92 

1.77 

.173 

CS 

14.35 

34.10 

.42 

.657 

AS 

.37 

1.18 

.32 

.728 

MC 

1.50 

2.07 

.73 

.484 

El 

1.13 

1.85 

.61 

.542 

^Statistically  significant  at  p  <  .03. 
^Statistically  significant  at  p  <  .01. 


Tabla  A.  Mun  and  Standard  Davlation  of  Scorat,  Kudar-Blchardaon  Foraula  2 
Rallabllity  Coafflc&ant  (KR20),  and  Nunbar  of  Exanlnaaa  (N),  for  tha  PC 

Subtaat  by  Group  and  Saaalon 


1.54  2.43  .637  56  -  12B  12.66  1.97  .595 
2.27  2.20  .627  56  -  12A  11.41  2.46  .633 
1.93  2.76  .750  55  -  13B  11.50  2.59  .653 

1.91  2.55  .675  57  -  13A  12.51  1.79  .470 
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12,  with  a  difference  of  1.41,  suggesting  that  no  practical  or  aeanlngful  sig¬ 
nificance  can  be  attached  to  this  outcome.  Therefore,  a  conclusion  of  no  im- 
portant  Fora  effect  was  drawn  for  the  paper-and-pencll  baseline  analysis. 

For  the  Session  effect,  the  NO  subtest  showed  an  Increase  in  aean  score 
of  1.37  froa  Session  1  to  Session  2,  CS  had  a  gain  of  5.75,  and  El  differed  by 
•28.  Therefore,  a  very  slight  NO  increase  and  a  somewhat  larger  CS  increase 
were  laplled  by  these  statistically  significant  aean  differences. 

The  PC  subtest  analysis  Indicated  a  significant  Version  by  Fora  by  Ses¬ 
sion  interaction.  However,  for  this  subtest,  the  range  in  aean  scores  froa 
the  lowest  to  the  highest  over  all  12  testing  conditions  was  only  1.30.  Be¬ 
cause  of  the  low  aagnitude  of  this  aean  difference,  this  interaction  was  in¬ 
terpreted  as  a  psychoastrically  nonaeanlngful  difference. 

Computer  versus  Paper-and-Pencll 

PC  subtest.  The  results  of  the  MAN OVA  for  the  PC  subtest  for  the  comput¬ 
er  and  paper-and-pencll  groups  are  reported  in  Table  10  (aeans  and  standard 
deviations  by  condition  are  in  Table  4).  The  statistically  significant  be- 
tween-subjects  effects  Include  the  aode  of  computer  administration  (Mode)  and 
coaputer  versus  paper-and-pencll  administration  (Medium).  No  significant  be- 
tween-subjects  interactions  were  revealed.  The  wlthln-subjects  repeated  mea¬ 
sures  factor  (Session)  was  found  to  be  significant,  as  well  as  the  Medium  by 
Session,  Version  by  Mode  by  Session,  Version  by  Medium  by  Session,  and  Mode  by 
Medlua  by  Session  interactions. 

The  follow-up  analysis  for  the  PC  subtest  began  with  the  Mode  effect. 

For  the  three  coaputer  screen  Modes,  the  mean  scores  were  10.32  for  CRT-1, 

8.46  for  CRT-2,  and  9.99  for  CRT-3;  the  corresponding  aean  differences  were 
1.86  raw-score  points  between  Modes  CRT-1  and  CRT-2,  .33  between  1  and  3,  and 
1.53  between  2  and  3.  This  indicates  that  screen  condition  2  was  different 
froa  1  and  3,  whereas  conditions  1  and  3  were  not  significantly  different  froa 
each  other. 

For  the  coaputer  versus  paper-and-pencll  Medlua  effect,  the  mean  differ¬ 
ence  within  Session  between  the  paper-and-pencll  group  (mean  -  12.19)  and  the 
coaputer  group  (aean  -  9.59)  was  2.60  points.  For  the  Session  effect,  the 
first  and  second  session  mean  difference  was  .82  points.  Upon  examination  of 
the  Medlua  by  Session  Interaction,  it  was  found  that  the  group  that  took  the 
coaputer  test  in  session  2  scored  2.15  points  higher  (mean  -  11.74)  than  the 
group  that  took  it  during  the  first  session  (aean  -  9.59).  For  the  paper-and- 
pencll  teats,  however,  the  second  session  mean  score  (mean  -  11.67)  was  only 
•52  lower  than  the  first  session  scores  (mean  -  12.19),  a  finding  of  little 
practical  Importance. 

The  Medlua,  Session,  and  Medlua  by  Session  effects  were  reevaluated  with 
the  low-scoring  second  presentation  Mode  group  removed.  This  reduced  the  Me¬ 
dlua  aean  difference  to  2.03  points,  and  Increased  the  Session  difference  for 
the  coaputer  groups  slightly  to  2.43  points. 


Of  the  three-way  interactions  of  Version  by  Mode  by  Session,  Version  by 
Medlua  by  Session,  and  Mode  by  Medium  by  Session,  all  revealed  only  minor  mean 
differences  of  less  than  1  point  across  conditions,  but  the  Mode  by  Session 
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Table  10.  Results  of  the  F-tests  of  Significance  froa  the 
MANOVA  for  the  Computer  Versus  Pape  r-and-Penc i 1  Adalnlstratlon 
PC  Subtest  Conditions  (N  •  664) 


Source  of  variation 

Degrees  of 
freedoa 

Mean 

square 

F 

P< 

Between  Subjects 

Version 

1 

.01 

.00 

.979 

Mode 

2 

218.66 

21.31 

.001** 

Medlua 

1 

590.51 

57.56 

.001** 

Version  x  Mode 

2 

.66 

.06 

.937 

Version  x  Medlua 

1 

.57 

.06 

.814 

Mode  x  Medlua 

2 

15.77 

1.54 

.216 

Version  x  Mode  x  Medlua 

2 

21.69 

2.11 

.122 

Within  Cells 

652 

10.26 

Within  Subjects 

Session 

1 

221.47 

65.88 

.001** 

Version  x  Session 

1 

.20 

.06 

.810 

Mode  x  Session 

2 

2.47 

.74 

.480 

Medlua  x  Session 

1 

531.81 

158.21 

.001** 

Version  x  Mode  x  Session 

2 

15.41 

4.58 

.011* 

Version  x  Medlua  x  Session 

1 

14.03 

4.17 

.041* 

Mode  x  Medlua  x  Session 

2 

90.48 

26.92 

.001** 

Version  x  Mode  x  Medlua  x  Session  2 

4.88 

1.4 

.235 

Within  Cells 

652 

3.36 

★Statistically  significant  at  p  <  .05. 
★★Statistically  significant  at  p  <  .01. 


interaction  was  a  aaanlngful  one  in  that  Mode  differences  existed  only  within 
one  of  the  two  Medina  conditions. 

Speeded  subtests.  The  results  of  the  MANOVA  for  the  speeded  NO  and  CS 
subtests  are  shown  in  Table  11  (mean  scores  by  condition  are  in  Table  5  for  NO 
and  Table  6  for  CS).  For  the  between-subjects  effects.  Medium.  Mode,  and  the 
Medlua  by  Mode  interaction  were  statistically  significant.  Within  subjects. 
Session.  Version  by  Session.  Medium  by  Session,  and  Medium  by  Mode  by  Session 
effects  were  found  to  be  statistically  significant. 

Table  12  contains  the  results  of  the  analyses  of  statistically  signifi¬ 
cant  effects  by  subtest.  Medlua  differences  were  indicated  for  the  NO  sub¬ 
test.  while  Mode  differences  were  observed  for  both  NO  and  CS.  The  between- 
subjects  Medlua  by  Mode  interaction  was  found  significant  for  the  CS  subtest. 
Within  subjects,  the  Session  factor  was  significant  for  both  NO  and  CS.  and 
all  of  the  Interactions  tested  were  significant  for  both  subtests  except  for 
Version  by  Session,  which  was  significant  for  NO  but  decidedly  nonsignificant 
for  CS. 

Exaaination  of  the  mean  scores  for  the  Medlua  effects  shows  that  the  po¬ 
pe  r-and-penc il  group  (aean  NO  -  37.79)  obtained  higher  scores  than  the  comput¬ 
er  group  (mean  NO  -  31.12)  in  Session  1  by  6.67  points  for  the  NO  subtest,  and 
8.22  (aean  CS  scores  of  60.96  and  52.74,  respectively)  for  CS;  both  differ- 


28 


’  k***  *  V*  w‘*  «  *  fc 


Table  11.  Pillai-Bartlett  Trace  V  Values  with  2  and  655 
Degrees  of  Freedan,  Approxlnate  F  Ratios,  and  Est lasted 
Significanca  Lavals  (p)  froe  tha  MAHOVA  for  tha  Conputer 
Varsus  Paper-and-Pencil  Administration  Conditions  for 
tha  NO  and  CS  Snbtasts  (N  ■  664) 


Source  of  variation  Trace  V 

F 

P< 

Between  Groups 

Version 

.00 

1.11 

.331 

Nedlun 

.01 

3.29 

.038* 

Node 

.23 

96.09 

.001** 

Version  x  Nedlun 

.00 

.82 

.439 

Version  x  Node 

.00 

.11 

.897 

Nedlun  x  Node 

.01 

3.30 

.038* 

Version  x  Nedlun  x  Node 

.00 

.04 

.965 

Vi  thin  Groups 

Session 

.06 

21.59 

.001** 

Version  x  Session 

.01 

3.54 

.030* 

Nedlun  x  Session 

.63 

556.01 

.001** 

Node  x  Session 

.00 

.08 

.925 

Version  x  Nedlun  x  Session 

.00 

.84 

.434 

Version  x  Node  x  Session 

.00 

1.43 

.241 

Nedlun  x  Node  x  Session 

.42 

237.07 

.001** 

Version  x  Nedlun  x  Node  x  Session 

.00 

.73 

.482 

♦Statistically  significant  at  p  <  .05. 
♦♦Statistically  significant  at  p  <  .01. 


ancas  wars  claarly  consequential,  svan  though  tha  AN OVA  analysis  found  tha  CS 
dlffaranca  aarglnally  statistically  nonsignificant.  In  Session  2,  tha  paper* 
and-penell  group  (aaan  ■  38.11)  out scored  tha  coapntar  group  (aaan  -  34.32)  by 
3.79  for  NO,  and  by  11.29  for  CS  (naans  ■  53.52  and  64.81).  Tha  CS  result  is 
claarly  substantiae.  The  Session  affect  overall,  with  a  1.77  point  gain  for 
NO  (naans  -  34.46  and  36.23),  and  2.32  points  for  CS  (naans  *  56.91  and 
59.23),  does  not  indicate  nuch  of  a  real  dlffaranca,  since  practice  affects 
would  be  expected  on  these  speeded  ltws. 

Upon  coaparlson  of  coaputer  screen  Node  differences,  it  was  found  first 
that  the  CRT-1  Node  condition  (naan  ■  34.02)  produced  higher  scoree  than  the 
CRT-2  condition  (nean  -  28.21)  in  Session  1  by  5.81  for  NO,  an  inportant  naan 
difference;  and  by  21.86  for  CS  (naans  -  71.89  and  50.03),  a  highly  inportant 
difference.  These  disparities  hardly  dlninlshed  for  the  computer  group  in 
Session  2,  with  an  NO  difference  of  5.42  (naans  -  37.03  and  31.61)  and  a  CS 
difference  of  18.45  (naans  -  74.04  and  55.59). 

Further  analyses  of  the  conputer  versus  papa r-and-pencil  nedlun  differ¬ 
ences  by  conputer  presentation  node  within  Session  1  revealed  that,  for  NO, 
the  CRT-1  condition  (nean  "  34.02)  showed  no  practical  difference  fron  the 
papa r-and-pencil  condition  (nean  ■  37.79),  but  that  the  CRT-2  condition  (nean 
-  28.21)  resulted  in  substantially  lower  scores  than  did  both  the  paper-and- 
pencll  condition  (nean  -  37.79)  and  the  alternate  conputer  Mode  (nean  - 
34.02).  The  sane  results  were  obtained  in  Session  2,  with  the  CRT-1  condition 


Table  12.  Results  of  Univariate  ANOVAs  for  Significant  Effects  Identified 
in  the  MAHOVA  for  the  Subtests  with  I  and  656  Degrees  of  Freedoa  (N  »  664) 


Effect  and 
subtest 

Mean 

Between 

squares 

VI thin 

F 

P< 

Medium 

HO 

685.89 

106.29 

6.45 

.011* 

CS 

782.94 

228.41 

3.43 

.065 

Mode 

HO 

3,006.37 

106.29 

28.23 

.001** 

CS 

40,997.32 

228.41 

179.49 

.001** 

Medlua  x  Mode 

HO 

14.35 

106.29 

.13 

.713 

CS 

1,145.51 

228.41 

5.02 

.025* 

Session 

HO 

1,028.29 

40.18 

25.59 

.001** 

CS 

1,784.08 

57.15 

31.22 

.001** 

Version  x  Session 

HO 

263.96 

40.18 

6.57 

.011* 

CS 

.88 

57.15 

.02 

.901 

Mediua  x  Session 

HO 

9,083.82 

40.18 

226.09 

.001** 

CS 

31,570.67 

57.15 

552.45 

.001** 

Mediua  x  Mode  x  Session 

HO 

2,256.84 

40.18 

56.17 

.001** 

CS 

27,114.36 

57.15 

474.47 

.001** 

*Statistlcally  significant  at  p  <  .05. 
*estatlstlcally  significant  at  p  <  .01. 


(eean  »  37.03)  not  being  different  froe  the  papa r-and-penc II  condition  (mean  - 
38.11),  but  with  both  of  these  conditions  revealing  naan  scores  much  higher 
than  those  of  the  CRT-2  condition  (asan  ■  31.60).  For  CS,  an  outcome  similar 
to  that  for  the  HO  subtest  was  obtained,  except  that  for  Session  1  the  CRT-1 
computer  presentation  condition  (aean  ■  71.88)  had  aean  scores  much  higher 
then  the  paper-and-pencil  (aean  -  52.74)  and  the  CRT-2  coaputer  presentation 
(mean  -  50.04)  conditions.  For  Session  2,  the  CRT-1  condition  (mean  -  74.04) 
was  again  auch  higher  than  either  the  peper-and-pendl  (aeen  -  53.52)  or  the 
CRT-2  (aean  -  55.58)  conditions. 

Although  a  significant  Version  by  Session  interaction  was  reported  by  the 
ANOVA  analysis  for  the  HO  subtest,  peirwise  comparisons  of  mean  differences 
found  no  statistical  significance.  The  largest  difference  discovered  was  that 
between  Version  A  in  Session  1  (aean  ■  33.64)  and  Version  A  in  Session  2  (aean 
-  36.30),  a  difference  of  only  2.66  points.  The  Version  B  means  were  35.26 
for  Session  1,  Increasing  to  36.13  in  Session  2.  For  the  Medium  by  Session 
interaction,  the  difference  between  paper-and-pencil  and  coaputer  administra¬ 
tion  scores  decreased  in  Session  2  for  HO  froa  6.67  to  3.79,  and  Increased  for 
CS  froa  8.22  to  11.28,  neither  by  a  significant  amount. 

Graphical  subtests.  Table  13  provides  the  results  of  the  MAHOVA  analysis 
for  the  AS,  MC,  and  El  graphical  subtests  (mean  scores  by  condition  for  these 
tests  are  in  Tables  5,  6,  7).  This  analysis  revealed  no  statistically  signlf- 
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Table  13.  Pillai-Bartlett  Trace  V  Values,  Degrees  of  Free doe, 
Approximate  F  Ratios,  and  Estlaated  Significance  Levels  (p)  from 
the  MAM OVA  for  the  Computer  Versus  the  Pape r-and-Penc 11 
Administration  Conditions  for  the  AS,  MC,  and  El  Subtests  (H  ■  664) 


Degrees  < 

Source  of  variation  Ttaca  V  Between 

of  freedom 
Within 

F 

P< 

Between  Groups 

Version 

ftjWYTY— 

3 

650 

.32 

.810 

Form 

6 

1,302 

1.44 

.196 

Medium 

3 

650 

2.34 

.072 

Version  x  Form 

.00 

6 

1,302 

.36 

Version  x  Medium 

.00 

3 

650 

1.03 

.379 

Form  x  Medium 

6 

1,302 

1.37 

.225 

Version  x  Form  x  Medium 

.00 

6 

1,302 

.48 

.827 

Within  Groups 

- 

Session 

.09 

3 

650 

20.24 

Version  x  Session 

.00 

3 

650 

.84 

.474 

Form  x  Session 

.01 

6 

1,302 

.82 

.552 

Medium  x  Session 

.04 

3 

650 

9.10 

.001** 

Version  x  Form  x  Session 

•00 

6 

1,302 

.35 

.908 

Version  x  Medium  x  Session 

.01 

3 

650 

1.88 

Form  x  Medium  x  Session 

.01 

6 

1,302 

1.45 

.191 

Version  x  Form  x  Medium  x  Session 

.01 

6 

1,302 

1.37 

.226 

Statistically  significant  at  p  <  .01. 


Table  14.  Results  of  Onlvarlate  ANOVAs  for  Significant 
Effects  Identified  In  the  MANOVA  for  the  AS,  MC,  and  El 
Subtests  with  1  and  652  Degrees  of  Freedom  (N  -  664) 


Effect  end 
subtest 

Mean 

Between 

squares 

Within 

F 

P< 

Session 

AS 

58.70 

2.19 

26.75 

.001** 

MC 

122.61 

2.93 

41.88 

.001** 

El 

17.45 

2.13 

8.21 

.004** 

Medium  x  Session 

AS 

3.89 

2.19 

1.77 

.184 

MC 

70.74 

2.93 

24.17 

.001** 

El 

21.86 

2.13 

10.28 

.001** 

^Statistically  significant  at  p  <  .01. 
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leant  effects  for  the  between-subjects  factors,  Version,  Vorn,  or  Hedlun. 
Significance  was  Indicated  for  the  repeated  neaaures  Session  effect,  and  the 
Hedlun  by  Session  Interaction.  Table  14  shows  the  results  of  the  univariate 
follow-up  analyses  for  the  statistically  significant  effects,  revealing  sig¬ 
nificance  for  all  three  graphical  subtests  on  the  Session  effect,  and  for  MC 
and  B1  on  the  Hedlun  by  Session  Interaction. 


I 
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Coaparlson  of  the  naans  by  condition  for  the  significant  effects  revealed 
snail  differences  In  naans  across  session  of  .42  for  AS,  .61  for  HC,  and  .23 
for  El,  certainly  not  of  any  psyehonetrlc  consequence.  For  the  Hedlun  by  Ses¬ 
sion  effect  for  HC  and  El,  differences  of  a  slallar  aagnltode  were  revealed, 
with  the  HC  coaputer  scores  Increasing  by  1.43  across  sessions  (naans  -  16.14 
and  17.57),  while  the  paper-and-pendl  scores  declined  by  .21  (naans  -  17.57 
and  17.21).  For  the  El  subtest,  coaputer  scores  gained  .66  across  sessions 
(naans  -  13.23  and  13.89)  while  the  paper-and-pencll  naan  score  was  .20  lower 
(naans  -  13.92  and  13.72),  all  relatively  snail  changes. 

Coaparlson  of  the  asans  for  the  MC  Session  1  Hedlun  effect,  specifically 
coaputer  (naan  -  16.14)  versus  paper-and-pencll  (naan  -  17.43)  presentation  of 
graphical  laage  ltans,  showed  a  nean  difference  of  1.29.  Further  inspection 
showed  that  the  largest  part  of  this  difference  was  attributable  to  Fora  11, 
where  the  paper-and-pencll  group  (nean  -  18.07)  out scored  the  coaputer  group 
(naan  -  15.82)  by  2.25.  For  Fora  12,  the  asans  were  16.17  for  the  coaputer 
group  end  16.88  for  the  paper-and-pencll  group,  for  a  difference  of  .71;  for 
Fora  13,  the  naans  were  16.42  and  17.32,  respectively,  for  a  difference  of 
.90. 

Sunnary 

Mo  important  psychonetrlcally  neaningful  significant  differences  were 
deaonstrated  for  the  paper-and-pencll  baseline  analysis  for  any  subtests,  ex¬ 
cept  for  a  practice  effect  on  CS.  For  the  coaputer  versus  paper-and-pencll 
equivalence  analysis,  the  PC  subtest  revealed  a  aajor  difference  In  the  Mode 
factor,  with  the  CRT-2  condition  resulting  In  inport ant  naan  differences  from 
the  CRT-1  and  CRT-3  conditions.  Even  with  the  effects  of  this  condition  re- 
aoved,  however,  differences  were  still  shown  between  the  computer  and  paper- 
and-pencll  presentation  nedla.  It  was  also  found  that  the  group  that  took  the 
coaputer  test  second  scored  higher  on  it  than  the  group  that  took  it  first. 
However,  the  group  that  took  the  paper-and-pencll  test  second,  did  not  obtain 
higher  scores  than  the  other  group's  first  session  paper-and-pendl  test. 

The  speeded  test  comparisons  revealed  that  for  NO,  the  CRT-2  presentation 
node  was  decidedly  inferior  In  performance,  while  the  CRT-1  condition  did  not 
differ  significantly  from  the  paper-and-pencll  results.  For  CS,  it  was  found 
that  the  CRT-1  computer  condition  resulted  in  higher  scores  then  did  either 
the  CRT- 2  or  the  paper-and-pencll  conditions,  which  did  not  differ  from  each 
other. 

The  analysis  for  the  graphical  subtests  revealed  no  psychonetrlcally 
meaningful  differences  for  any  effects  for  any  subtests,  including  the  first 
session  Medium  affect  for  the  MC  test  which  addresses  directly  the  question  of 
differences  due  to  paper-and-pencll  versus  computer  presentation  of  graphical 

laage  test  Items. 
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fallibility  Analysis 


jytjgtggt  Corralatlona 


Tibia  15  shows  test-rsteat  correlations  (for  CS,  AS,  MC  and  El)  or  altar- 
nata  forna  ratast  corralatlona  (for  PC  and  NO)  for  aach  subteat.  For  tha 
apaadad  snbtaats  (MO  and  CS),  all  tha  corralatlona  for  Groupa  1  and  2  for  both 
coapotar  adnlnlatratlon  aodaa  vara  algnlflcantly  lovar  (at  tha  .05  laval)  than 
thoaa  froa  Group  3,  ax cap t  for  tha  NO  anbtaat  In  Group  2  coaputar-adalnlatarad 
Moda  1.  Thaaa  significant  dlffarancaa  In  ratast  corralatlona  vara  fairly 
large,  tha  saallast  balng  .20  for  CS  (.65  Tarsus  .85)  In  Group  2  Mods  2, 
Incraaslng  to  .36  for  NO  (.33  versus  .69)  In  Group  1  Node  2. 

Tsbla  15.  Tast-Eatast  Corralatlona  (r),  for 
Subtasts  CS,  AS,  MC,  and  El  and  Altamata 
Foras  Ratast  Corralatlons  for  PC  and  NO 
Subtasts  and  Nuaber  of  Exaalnaas  (N) 
by  Group  and  Moda 


Subtast 

8  aoda/ 
version 

Group  1 

Group  2 

Group  3 

r 

N 

r 

N 

r 

N 

PC 

1 

.42 

114 

.75* 

109 

.49 

112 

2 

.45 

109 

.49 

112 

3 

.33 

109 

.79* 

111 

NO 

1 

.44* 

168 

.65 

168 

.69 

108 

2 

.33* 

164 

.50* 

164 

CS 

1 

.57* 

168 

.56* 

168 

.85 

108 

2 

.60* 

164 

.65* 

164 

AS 

11 

.91 

114 

.88* 

109 

.93 

112 

12 

.93 

109 

.88* 

112 

.94 

108 

13 

.89* 

109 

.95 

111 

.95 

113 

MC 

.81* 

332 

.87 

332 

.88 

333 

El 

.79* 

332 

.89 

332 

.85 

333 

Rota.  Group  3  PC  corralatlon  Is  based  on  exaalnees 
l&Mmg  Fora  11  only,  and  tha  NO  and  CS  corralatlons 
era  based  on  exaalnaas  taking  Fora  12  only,  in 
correspondence  vlth  tha  foras  adalnlstered  to  Groups 
1  and  2. 


♦Indicates  that  the  Group  1  or  Group  2  correlation 
Is  significantly  different  from  the  Group  3  corre¬ 
lation  at  tha  .05  laval. 


Tha  graphical  aubeaata  (AS,  NC,  and  El)  shoved  a  aiallar  pattarn  of  teat- 
rataat  corralationa.  For  aach  graphical  subteat,  tha  Group  1  corralationa 
vara  significantly  lover  than  thoaa  froa  Group  3,  although  for  AS  only  Fora  13 
ylaldad  algnlficantly  lovar  corralationa,  Tha  only  Group  2  test-reteat  corra¬ 
lationa  algnlflcantly  lovar  than  Group  3  vara  froa  AS,  Forma  11  and  12.  Al¬ 
though  thaaa  raliabllltlaa  (taat-rataat)  vara  atatlatlcally  algnlflcantly  lov¬ 
ar  for  tha  groupa  In  which  ona  taat  adalnlatration  waa  coaputarlaad,  tha  actu¬ 
al  dlffarancaa  In  taat-rataat  corralationa  vara  not  vary  large,  ranging  from 
•05  (AS  in  Group  2,  Form  11)  to  ,07  (MG  in  Group  1), 

Tha  pattarn  of  taat-rataat  corralationa  waa  aoaavhat  different  for  sub- 
teat  PC  aa  there  vara  no  dlffarancaa  In  taat-rataat  corralationa  between 
Groupa  1  and  3  that  reached  aignlflcanca  at  tha  ,05  level,  Tha  Group  2  PC 
taat-rataat  corralationa  were  algnlflcantly  higher  than  Group  3  for  Modes  1 
and  3,  Thaaa  ware  tha  only  taat-rataat  corralationa  for  any  of  tha  subtaata 
that  vara  algnlficantly  higher  In  either  of  tha  coaputer  adalnlatration  groupa 
(1  and  2)  than  In  tha  paper-and-pendl-only  Group  3  (.75  and  ,79  versus  ,49). 

Internal  Consistencies 


Mean  KR-20s  for  each  non- speeded  subtest  by  Group  and  Session  are  In 
Table  16  (KR-20s  for  each  group  and  condition  are  in  Tables  4  through  9), 
Tables  4  through  9  show  that  the  differences  In  KR-20s  within  calls  vara  of 
tha  same  magnitude  aa  any  difference  found  betveen  cells;  thus,  tha  data  ware 
studied  aa  naans  Instead  of  as  Individual  values  froa  separate  teat  adminis¬ 
trations. 

Table  16.  Average  Kudar-Rlchardson  Formula  20  Reliability  (KR20) 


for 

Subteats  PC,  AS, 

,  MC,  and  El  by  Experimental  Group  and  Suasion 

Group 

Suasion  1 

Session  2 

PC 

AS 

MC 

El 

PC 

AS 

MC 

El 

Computer 

Paper-and-Pencll 

Group  1 

.558 

.817 

.758 

.728 

.674 

.824 

.764 

.720 

Paper-and-Pendl 

Coaputer 

Group  2 

.658 

.808 

.714 

.694 

.596 

.800 

.711 

.699 

Paper-and-Pendl 

Paper-and-Pencll 

Group  3 

.727 

.806 

.748 

.697 

.694 

.794 

.740 

.692 

Mote.  For  the  PC  subtest  In  the  coaputer  conditions,  only  Modes  1  and  3 
were  Included,  due  to  the  extreme  Mode  2  differences.  In  the  paper-and- 
pencll  conditions,  only  Form  11  was  included  so  as  to  correspond  with  the 
computer  conditions. 

The  lowest  mean  reliabilities  for  both  computer  administration  (.558  and 
•658)  and  paper-and-pencil  (.727)  were  obtained  for  the  PC  subtest,  as  expect- 
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•d,  due  to  Its  short  length.  Also,  tho  largest  differences  In  assn  KR-20s 
ecross  nedia  of  administration  for  any  subtest  were  found  for  the  reliabili¬ 
ties  of  PC,  with  differences  of  .100  (Cll  versus  P21)  and  .169  (Cll  versus 
P31).  The  PC  subtest  slso  showed  the  only  lerge  lncreese  in  naan  KR-20  across 
sessions,  with  the  Group  1  naan  KR-20  increasing  fron  .558  in  Session  1  to 
•674  in  Session  2. 

Subtest  AS  yielded  such  aore  consistent  naan  KR-20  values,  with  e  range 
fron  .794  (Group  3,  Session  2)  to  .824  (Group  1,  Session  2).  This  .03  range 
is  striking  in  that  both  values  ere  fron  Session  2  paper-a ad-pencil -admini¬ 
stered  tests,  inplying  that  the  conputer  versus  paper-and-pendl  comparisons 
were  nore  equal,  with  only  .009  (Cll  versus  P21)  end  .011  (Cll  versus  P31) 
differences  in  Session  1  and  .024  (P12  versus  C22)  end  .006  (C22  versus  P32) 
differences  in  Session  2. 

The  Session  1  end  2  aean  KR-20  differences  were  all  smaller  than  .01  for 
subtest  MC.  This  corresponds  with  the  pattern  found  for  subtest  AS,  where  the 
largest  difference  was  a  .012  decrease  fron  Session  1  to  Session  2.  Within 
Session  1,  there  was  e  larger  aean  KR-20  difference  for  NC  between  the  two 
paper-and-pencil  adninlstratlons  (P21  versus  P31,  .03)  than  between  Cll  and 
P31  (.01),  though  ell  the  differences  were  very  saall. 

The  aean  KR-20  values  for  subtest  El  conformed  to  the  general  pattern 
found  for  the  other  graphical  subtests,  indicating  no  effect  on  Internal  con¬ 
sistency  for  conputer  versus  paper-and-pencil  test  adalnlstretlon.  As  found 
for  MC,  there  was  no  conputer  administration  effect  within  examinees  for  El, 
with  ell  differences  in  naan  KR-20*  ecross  sessions  being  .008  or  less.  In 
both  sessions,  the  naan  KR-20  was  higher  in  Group  1  than  in  the  other  groups, 
with  the  overall  range  being  .692  (P32)  to  .728  (Cll). 


Analysis  of  Structure 


Across  Subtests 


For  the  calculation  of  the  between-subtest  correlation  and  covariance 
matrices  fron  the  computer  administrations,  only  scores  obtained  fron  exami¬ 
nees  taking  Modes  1  and  3  for  the  PC  subtest  and  Mode  1  of  the  MO  and  CS  sub¬ 
tests  were  Included.  The  MAN0VA  analysis  showed  that  the  Mode  2  scores  for 
PC,  NO,  end  CS  were  much  different  then  both  Mode  1  and  Mode  3  in  terms  of 
subtest  naans  end  variances,  thus  their  exclusion  fron  this  analysis.  Table 
17  shows  the  correlations  between  subtast  scores  by  Group  and  Session  which 
were  analyzed  by  principal  factor  analysis  to  yield  the  eigenvalues  shown  in 
Table  18.  The  elganstructure  is  fairly  similar  within  each  cell  (Group  and 
Session  combination),  with  e  large  first  factor  capturing  between  63. 41  (P21) 
end  69.8Z  (P32)  of  the  connon  variance.  All  cells  seen  to  confora  to  a  two- 
factor  solution  except  Cll,  for  which  three  factors  are  indicated.  Cell  Cll 
is  the  only  one  where  the  third  eigenvalue  is  positive,  with  the  third  factor 
accounting  for  9.9Z  of  the  connon  variance.  The  third  factor  is  present  at 
the  expense  of  Factor  2  which  accounts  for  13. 3Z  less  connon  variance  than  the 
Factor  2  in  any  other  cell  in  Session  1. 


Table  19  gives  the  factor  loadings  by  Group  and  Session  froa  the  princi¬ 
pal  factor  analyses.  Factor  1  loads  highest  on,  and  is  defined  by,  the  graph¬ 
ical  subtests  (AS,  MC,  El)  in  every  cell  except  P31,  where  PC  loads  higher 
than  AS  (.572  versus  .542).  There  are  also  substantial  loadings  for  PC  on 


Table  17.  Intercor relation*  of  Subtest  Scores 
by  Group  for  Session  1  (Upper  Triangle)  end 
Session  2  (Loner  Triangle) 


Subtest 

PC 

NO 

CS 

AS 

MC 

El 

Group  1  (N-171  Session  la; 

N-332  Session  2) 

PC 

— 

.36 

-.06 

.08 

.30 

.32 

NO 

.14 

— 

.27 

-.05 

.19 

.11 

CS 

.13 

.56 

— 

.08 

.18 

.10 

AS 

.25 

-.05 

.01 

— 

.59 

.49 

MC 

.32 

.00 

.08 

.52 

— 

.56 

El 

.34 

-.02 

.00 

.56 

.59 

— 

Group  2  (N-332  Session  1;  ] 

N-167  Session  2a) 

PC 

_ 

.20 

.22 

.16 

.25 

.28 

NO 

.15 

— 

.62 

-.04 

.12 

-.02 

CS 

.27 

.41 

— 

.11 

.21 

.10 

AS 

.24 

-.17 

.01 

— 

.54 

.57 

MC 

.34 

-.01 

.11 

.56 

— 

.55 

El 

.40 

-.09 

.00 

.52 

.54 

— 

Group  3  (N-333 

,  both 

sessions) 

PC 

— 

.39 

.34 

.21 

.36 

.35 

NO 

.26 

— 

.63 

-.03 

.16 

.07 

CS 

.27 

.57 

— 

.02 

.21 

.09 

AS 

.22 

-.03 

.05 

— 

.53 

.55 

MC 

.41 

.18 

.28 

.52 

— 

.52 

El 

.37 

.11 

.16 

.58 

.57 

— 

Note.  Only  factors  with  eigenvalues  greater  than  zero 
are  presented. 


aOnly  computer  Modes  1  and  3  for  PC,  and  Mode  1  for  NO 
and  CS,  were  Included  since  Mode  2  for  PC  and  Mode  2 
for  NO  and  CS  were  found  to  be  significantly  different 
In  the  MANOVA  analysis. 


Factor  1  for  cells  C22  (.488)  and  P32  (.521),  crossing  both  Session  and  Medl- 
ua. 


The  second  factor  Is  a  little  cleaner,  with  Its  highest  loadings  found  on 
the  speeded  subtests  (NO  and  CS)  In  every  cell  except  Cll,  where  PC  and  AS 
have  stronger  loadings  than  CS  (-.432  versus  .350).  In  Group  1,  Session  1, 
both  PC  and  AS  have  stronger  loadings  than  CS  (.357  and  -.347  versus  .184). 

In  Group  3  there  Is  also  a  substantial  loading  on  Factor  2  for  subtest  AS  In 
each  session  (-.443  and  -.385). 


Assisting  that  a  third  factor  Is  necessary  to  adequately  explain  the  data 


Table  18.  Eigenvalues  and  Parcantaga  of  Coaaon 
Variance  Accounted  for  fron  Principal  Factor  Analyses 
of  Subtast  Scores  by  Group  and  Session 


Session  1 _  _ Session  2 


Group  and 
factor 

Eigen¬ 

value 

Z  of 

variance 

Eigen¬ 

value 

TTf 

variance 

Group  1: 

Computer  (N-171)4 

Paper-and-Pendl  (N-332) 

1 

1.740 

67.8 

1.719 

65.5 

2 

.572 

22.3 

.906 

34.5 

3 

Group  2: 

.255  9.9 

Paper-and-Pencil  (N-332) 

Computer  (N«167)a 

1 

1.740 

63.4 

1.732 

69.3 

2 

1.006 

36.6 

.768 

30.7 

Group  3: 

Paper-and-Pencll  (N-333) 

Paper-and-Pendl  (N-333) 

1 

1.885 

64.4 

1.956 

69.8 

2 

1.043 

35.6 

.846 

30.2 

Note.  Only  factors  with  eigenvalues  greater  than  zero  are 
presented. 


aOnly  computer  Modes  1  and  3  for  PC*  and  Mode  1  for  NO  and  CS, 
were  Included  since  Mode  2  for  PC  and  Mode  2  for  NO  and  CS 
were  found  to  be  significantly  different  in  the  MANOVA  analysis. 


In  cell  Cll*  It  is  defined  by  low  to  aoderate  loadings  on  PC  (->.289)  and  CS 
(.389).  The  coaaunallty  estimate  for  CS  In  cell  Cll  (.220)  Is  lower  than  In 
any  other  cell*  the  next  lowest  being  .337  In  the  other  computer-administered 
cell  C22. 

Table  20  summarizes  the  covariance  structure  and  confirmatory  factor 
analysis  results.  The  first  model  tested  was  the  equality  of  the  across- 
subtast  covariance  matrices  within  sessions.  In  both  sessions  this  model  was 
rejected  as  not  fitting  the  data*  with  X2  -  104.57  and  X2  -  75.47  with  42  de¬ 
grees  of  freedom.  A  check  on  the  sensitivity  of  the  procedure  was  run  testing 
the  equality  of  the  across-subtest  covariance  matrices  for  only  the  paper-and- 
pencll  administrations  within  each  session.  This  model  was  not  rejected  in 
either  session  (X2  •  22.42  and  X2  -  29.91*  with  21  degrees  of  freedom  each)* 
indicating  that  the  covariance  matrices  for  computer  administration  sessions 
were  not  equal  to  those  fron  paper-and-pencll  testing  sessions. 

The  next  model  tested  a  less-strong  equality  that  the  factor  pattern  was 
invariant  for  each  group  within  sessions.  The  principal  factors  analyses  sug¬ 
gested  that  either  two  or  three  factors  were  present;  thus*  the  first  tests 
assumed  the  presence  of  three  correlated  factors  for  each  group.  Assuming 
that  this  model  was  not  rejected,  subsequent  models  would  test  the  equality  of 
the  factor  loadings,  error  variances  and  covariances,  and  factor  covariances* 


Table  19.  Factor  Loading!  and  CoMunalities  (h  )  iron  Principal 
Factor  Analyses  of  Subtest  Scores  by  Group  and  Seaalon 
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Table  20.  Sunary  Table  for  Tatting  Models  of  Factor  Equality 
Vi thin  Session,  Showing  Model,  Chi-Square  Value  (X2),  Degrees 
of  Freed on  (df),  Goodness  of  Fit  Index  (GF1),  Root-Mean  Square 
Residual  (RMR),  end  Eat lasted  Probability  (p) 


Hypothesis 

x2 

df 

GFI 

RMR 

P< 

$11  “  $21  “  $31 

104.57 

42 

.977 

2.206 

.001** 

$21  “  $31 

22.42 

21 

.989 

2.090 

.374 

$12  *  $22  “  $32 

75.47 

42 

.985 

3.513 

.001** 

$12  ”  $32 

29.91 

21 

.984 

4.632 

.094 

A. 

S«  '  N.  ■  S, 

66  not 

positive  definite  for 

Cll 

A. 

^*12  A*22  ^*32 

56.58 

21 

.967 

2.297 

.001** 

B. 

A-  -  A  ■  A  . 

X11  X21  *31 

*  not  positive  definite  for 

C22 

B. 

A_  •  A_  ■  A_ 
X12  *22  x32 

*,  66  not  positive  definite  for  Cll 

C. 

A*  “A-  -  A_ 

X11  X21  X31 

157.54 

24 

.940 

2.239 

.001** 

♦♦Statistically  significant  at  p  <  .01. 


Mote.  A  is  a  as del  of  the  factor  pattern: 


Ax 


'0  0  x" 

0X0 
0X0 
X  0  0 
X  0  0 
X  0  0 


B  is  a  nodal  of  the  factor  pattern: 


Ax 


"0  0  X* 
0X0 
0X0 
X  0  0 
X  0  X 
X  0  X 


C  is  a  nodal  of  the  factor  pattern: 


A 


x 


‘0  x' 
0  X 
0  X 
X  0 
X  0 
.X  0. 
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providing  an  Indication  of  the  degree  of  equality  of  the  factor  structures 


Only  for  Session  2  did  LISKEL  converge  to  Meaningful  paraaeter  estimates 
while  testing  a  three-factor  model.  This  model  (Factor  1  -  AS,  MC,  El;  Factor 
2  ■  HO,  CS;  Factor  3  -  PC;  zeroes  elsewhere)  produced  e  significant  chi-square 
(X2  »  56.58,  21  degrees  of  freedom)  indicating  non-fit.  In  Session  1,  the 
matrix  theta-delta  was  not  positive  definite  for  Group  1  (Cl 1 ) ,  Indicating 
that  the  model  did  not  fit  the  data  for  that  cell. 

A  second  three-factor  pattern  was  hypothesized,  allowing  two  more  load¬ 
ings  to  vary  (Factor  1  -  AS,  MC,  El;  Factor  2  -  HO,  CS;  Factor  3  -  PC,  MC,  El; 
zeroes  elsewhere),  which  lessened  the  pressure  to  fit  many  zero  loadings.  For 
this  model,  LISSEL  did  not  converge  to  meaningful  paraaeter  estiaetes  In 
either  session.  The  Session  1  results  found  both  matrices  phi  and  theta-delta 
to  be  non-positive  definite  for  Group  1  (Cll).  In  Session  2,  phi  was  not  pos¬ 
itive  definite  for  cell  C22  (Group  2).  Again,  the  non-ma aningf ul  estimates  of 
these  matrices  Imply  that  the  model  did  not  fit  the  data  within  those  cells. 

Next,  a  two-factor  pattern  model  (Factor  1  -  AS,  MC,  El;  Factor  2  -  PC, 
NO,  CS;  zeroes  elsewhere)  was  tried  for  Session  1,  in  case  three  factors  were 
not  necessary  to  explain  the  data  adequately.  This  model  fit  the  data  less 
well  than  the  most  restrictive  covariance  equality  model  (x2  -  104.57,  df  - 
42),  with  e  X2  value  of  157.54  with  24  degrees  of  freedom.  The  implication  of 
this  test  Is  that  neither  a  two-factor  model  nor  a  three-factor  model  fit  the 
Session  1  data.  Thus,  the  reason  for  the  non-fit  of  the  three-factor  model 
could  not  be  the  specification  of  too  many  factors. 

Within  Subtest 


Table  21  shows  eigenvelues  and  the  percentage  of  common  variance  account¬ 
ed  for  from  the  within-subtest  principal  factors  analysis  by  Form  (11,  12  and 
13)  and  Medium  of  administration.  Appendix  A  shows  the  factor  loadings  for 
the  first  five  principal  factors  from  the  same  analyses.  Only  non-speeded 
subtests  AS,  MC,  and  El  ware  factor  analyzed,  because  it  was  necessary  to  com¬ 
bine  the  data  from  examinees  talcing  both  Versions  A  and  B  in  order  to  obtain 
enough  examinees  to  meet  factor  analysis  requirements.  Subtest  PC  contains 
different  items  In  Versions  A  and  B,  making  such  a  combination  impossible. 

Due  to  the  finding  of  significant  Session  effects  In  the  MANOVA  analysis,  all 
analyses  were  performed  within  Session  1,  with  Group  1  (computer  administra¬ 
tion)  being  compared  against  Group  3  (paper-and-pencll  administration). 

There  were  no  major  trends  across  subtests  or  forms,  or  large  factor 
structure  differences  for  the  subtests,  under  different  media  of  administra¬ 
tion  conditions.  The  factor  structures  of  each  fora  of  each  subtest  under 
both  administration  media  were  adequately  described  by  one-factor  solutions. 
The  large  number  of  eigenvalues  greater  than  1.0  found  for  each  subtest  is 
likely  an  indication  of  the  small  examinee-per-ltea  ratio  for  each  factor 
analysis  (4:1  or  5:1).  The  first  factors  accounted  for  43. 8Z  (MC  Form  13, 
computer-administered)  to  59. 42  (AS  Form  13,  computer-administered)  of  the 
common  variance  in  each  subtest.  These  first  factors  are  more  than  twice  as 
large  as  the  second  factors  in  every  case,  suggesting  the  interpretation  of  a 
one-factor  solution.  All  subtests,  however,  showed  a  trend  for  the  first  fac¬ 
tor  to  be  larger  (l.e.,  account  for  more  common  variance)  under  computer  ad¬ 
ministration  than  for  paper-and-pencll  administration.  These  differences 
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Table  21.  Eigenvalues  (E)  end  Percentage  of  Coneon  Ver 
Accounted  for  <%),  by  the  First  Five  Principal  Factors 


7 


m  m 

8  8  5 

■h  -a 

£ 


•O  M 

£1 

&s 

-h  m  u 

»-«  Ou 


h 

•  fl  2 
£•  8 


N  N  >0  ■»  vO 
..... 
CM  <9  "M  ^ 

H  H  H 


O  <  H  * 

H  H  40  Is 

<p4000 


<ONnH 
..... 
OiN  « 

m  «m 


ifl  n  ifl  oi  O' 
OOOlS'fl 
..... 
iflHOOO 


m  mt  m  cn  cm 
..... 
in  -«r  cn  a\  oo 


..... 

•tf  -H  o  o  o 


*o  oi  m  eo 
..... 
vo  m  cm  i>« 

*n  •— i 


<  N  N  O'  ^ 
o-tf  H<o  « 
..... 
WHHOO 


>9  H  CM  H  (M 
..... 
O  Ifl  n  (M  ffl 
m  H  H  H 


O'  9  O'  H  O' 
N  H  O'  O'  'O 
..... 
m  -h  o  o  o 


BO*  NO  N 
..... 
9  ifl  O  O'  O' 


BNBO'O 
W  CM  00  N  N 
..... 
<o  -n  ©  ©  © 


h  cm  cn  ■*  in 


in  cn  o\  O'  <o 


h  O'  m  h  co 

rt  H  O'  oo  N 
..... 
PtHOOO 


00  «  B  CM  N 
..... 
cn  >o  >v  on 


>o  -o  n  n  <n 
r.  o  O'  oo  n 
•  •  .  •  « 
CM  H  O  O  O 


o  m  «  ao  eo 
..... 
om  m  >m  o  O' 

m  «-h  •-*  r-i 

Mn  O'  o  cn 
oo  H  co  oo  N 
..... 
cn  «m  o  o  o 


OHCMON 

..... 

■OPlNHOl 


OinBHN 
in  OO  N  N  >o 
..... 
cn  o  o  o  o 


in  b  m  M>  n 
..... 
B'O'ONO 
-O  -4  »-• 


n  in  n  vo  cn 

H  H  O'  00  N 
..... 
cn  -h  o  o  o 


ao  O'  o  m  co 
..... 
(O  H  H  o  O' 


O'  cm  m  h  m 
cn  o'  oo  oo  n 
..... 
<00000 


h  om  cn  <o  m 


•O  00  N-  CM 
.... 
n  >o  cn  cn 


cn  <o  o  n 
a  b  n  <o 
.  .  .  . 
CM  O  O  O 


cm  ao  cn  o 
.  .  .  . 
cn  r*  cm  O' 

in  -h  -h 


O'  <o  cm  cn 
O  O  p*  m 
.  .  .  . 
cnnoo 


O  N  CM  00 
.  .  .  . 
n  <  cm  O' 
in  h  h 


<  CM  «  N 
■9  O'  N  B 
.  .  .  . 
cn  o  o  o 


in  cn  o'  o> 
.  .  .  . 
p-  <o  O  ao 
in  .-h  —i 


O'  h  cn  h 
CM  ao  b  m 
.  .  .  . 
cn  o  o  o 


9  O'  CM  lx 
•  «  •  « 
o-ccn  rt 
Bhhh 


cn  fx*  b  co 

O'  OO  S  B 
.  .  .  . 
CM  O  O  O 


MB  N  N 
.  .  .  • 
O  Px  CM  O 
m  —I 


B  OO  N  O' 
OONM) 
.  .  .  « 
cn  -n  o  o 


h  cm  cn  9  >n 


0.51  8. A  0.57  9.8  0.48  8.4  0.52  8.3  0.44  7.6  0.45  8.9 


ranged  froa  11.3Z  in  MC  Font  11  to  1.1Z  in  AS  Form  12.  Thar*  was  no  differ¬ 
ence  in  El  Form  11,  end  there  ties  a  1.7Z  difference  In  the  other  direction  for 
MC  Font  13,  but  these  were  the  only  exceptions. 


The  only  large  difference  In  computer-  versus  paper-and-pencil-admlnl- 
stared  subtest  factor  structure  occurred  for  subtest  MC,  Fore  11.  For  comput¬ 
er  administration,  Factor  1  accounted  for  56.8Z  of  the  common  variance,  but 
for  the  paper-and-pendl  administration.  Factor  1  captured  only  45. 5Z,  causing 
an  11.3Z  difference.  This  difference  was  not  matched  in  Forms  12  or  13  of 
subtest  MC;  in  fact,  for  Form  13,  tha  paper-and-pendl  Factor  1  accounted  for 
more  common  variance  than  did  tha  Factor  1  from  the  computer  administration 
(45.5Z  versus  43.8Z). 


Item  Analysis 

Conventional  item  statistics  (proportion  correct,  po in t-bi serial,  and 
bl serial  item-test  correlations)  for  each  item  in  each  nonrspeeded  subtest 
(PC,  AS,  MC  and  El)  by  Version  (A  or  B)  and  Medium  (computer  or  paper-and- 
pendl)  are  shown  in  Appendix  B.  There  were  no  statistical  analyses  performed 
at  the  item  level  to  compare  particular  values  of  these  statistics,  due  to  the 
mostly  nonsignificant  and  inconsequential  differences  found  for  the  Medium 
effect  in  the  MANOVA  analyses. 

The  important  question  of  possible  differences  in  graphical  content  items 
across  administration  media  was  addressed  partially  through  the  subtest-level 
MANOVAs.  The  distribution  of  graphical  items  allowed  the  comparisons  of  sub¬ 
test  MC  across  administration  media  to  substitute  for  an  item-level  analysis. 
In  fact,  every  subtest-level  analysis  performed  on  subtest  MC  could  be  used  as 
a  method  to  compare  graphical  versus  non-graphlcal  content  items.  By  compar¬ 
ing  computer  administration  versus  paper-and-pendl  administration  differences 
for  MC  versus  AS  and  El,  information  on  graphical  items  was  obtained.  The 
results  of  these  comparisons  were  that  differences  in  subtest  MC  across  admin¬ 
istration  media  were  not  significantly  greater  than  the  computer  versus  paper- 
and-pencil  differences  exhibited  by  AS  or  El.  The  wlthln-subtest  factor  anal¬ 
yses  also  showed  no  greater  differences  in  the  factor  structure  of  MC  across 
administration  media  than  were  found  for  subtests  AS  and  El. 


IV.  DISCUSSION 

The  purpose  of  this  study  was  to  determine  the  extent  of  equivalence  of 
measurement  properties  for  a  battery  of  ASVAB  subtests  under  conditions  of 
computer  versus  paper-and-pendl  administration.  The  subtests  selected  for 
study— PC,  HO,  CS,  AS,  MC,  end  El— were  those  that  presented  particular  prob- 
lema  for  computer  presentation. 

The  paper-and-pendl  baseline  analysis  was  performed  to  yield  e  lower 
bound  against  which  computer  versus  paper-and-pencil  differences  could  be 
Judged,  both  within  end  between  examinees.  The  only  statistically  significant 
effects  found  were  those  for  Form  on  the  AS  subtest  and  Session  for  the  NO, 

CS,  end  AS  subtests.  However,  the  statistically  significant  mean  differences 
observed  were  small  and  not  psychomatrlcally  meaningful,  thus  providing  an 
acceptable  basis  for  judging  comparisons  between  computer  and  paper-and-pencil 
conditions. 


Paragraph  Comprehension  Subfit 


Tha  PC  sub test,  choaan  dua  to  It a  paragraphrlength  Itaaa  and  multiple 
It ana  par  paragraph,  ahowad  dlffarancaa  In  measurement  propartlaa  acroaa  tha 
medium  of  praaantation  and  within  tha  thraa  aodaa  of  coaputar  administration, 
with  ona  Mode  (CRT-2)  dif faring  aignlflcantly  froa  tha  othar  two  (CRT-1, 
CRT-3)*  Tha  first  Hoda  (CRT-1)  dlaplayed  tha  paragraph  In  ona  scrolling  flald 
whlla  tha  Itaaa  appaarad  aequentlally  In  a  saparata  nonscrolling  flald  banaath 
It*  Each  Itaa  was  arasad  bafora  tha  naxt  ona  appaarad,  and  tha  axaalnaa,  un- 
abla  to  ratrlava  tha  ltaa,  could  tharafora  procaad  only  in  a  forward  dir act Ion 
through  tha  tast.  Tha  sacond  coaputar  adalnistratlon  Mods  (CRT-2)  contained 
aach  paragraph  and  all  relevant  questions  in  a  single  scrolling  field*  In 
this  node,  the. entire  screen  was  available  for  viewing  tha  paragraph,  with 
questions  appearing  after  tha  final  line  of  text  as  tha  paragraph  was  scrolled 
up  tha  screen,  allowing  tha  axaalnaa  to  nova  back  and  forth  within  each  para¬ 
graph.  For  Sessions  1  and  2,  this  condition  resulted  In  naan  scores  signifi¬ 
cantly  lower  than  both  tha  paper-and-pencll  condition  and  tha  othar  two  CRT 
conditions,  suggesting  axaalnaa  confusion  or  disorientation  arising  froa  this 
particular  screen  foraat.  Another  possibility  Is  that  because  whole  para¬ 
graphs  and  corresponding  ltaas  never  appaarad  on  tha  screen  simultaneously,  a 
aaaory  component  was  Introduced  and  became  sore  Important  In  this  condition 
than  In  tha  others*  Tha  final  coaputar  condition  (CRT-3)  contained  saparata 
scrolling  fields  for  both  tha  paragraphs  and  thalr  related  ltaas,  and  provided 
an  answer-sheet  type  of  display  at  tha  top  of  tha  screen,  allowing  exaalnees 
to  aonltor  thalr  progress  through  tha  tast.  This  condition  also  enabled  exam¬ 
inees  to  return  to  any  paragraph  and  to  Change  thalr  response  to  any  item  at 
any  tine  during  tha  teat*  In  both  sessions,  this  condition  resulted  In  aaan 
scores  almost  Identical  to  those  for  tha  CRT-1  condition. 

Clear  differences  ware  demonstrated  within  Session  1  between  all  three 
CRT  conditions  and  tha  paper-snd-pencil  administration  condition.  In  Session 
2  however,  equivalent  scores  ware  obtained  for  the  paper-and-pencll,  CRT-1, 
and  CRT-3  conditions,  with  the  aaan  paper-and-pencll  score  being  fairly  con¬ 
stant,  while  the  scores  for  all  thrae  CRT  groups  were  significantly  larger 
compered  to  Session  1.  Only  the  CRT-2  condition  still  yielded  significantly 
lower  scores  than  paper-and-pencll  In  Session  2.  This  finding  suggests  that 
those  who  took  the  paper-and-pencll  PC  test  first,  followed  by  the  coaputar  PC 
test,  aay  have  benefit tad  froa  practice  effects  in  the  second  session,  imply¬ 
ing  that  a  more  extensive  practice  sequence  and  perhaps  a  more  detailed  In¬ 
struction  set  preceding  the  adalnistratlon  of  the  computerized  PC  test  nay  be 
appropriate.  In  light  of  the  absence  of  comparable  differences  on  the  other 
subtests,  another  possibility  may  be  that,  due  to  the  combination  of  lack  of 
familiarity  with  the  computer  medium  and  the  complexity  of  the  PC  subtest,  the 
first  subtest  administered  In  the  sequence  produced  a  heightened  level  of  anx¬ 
iety  which  attenuated  test  scores  for  those  tested  by  computer  in  the  first 
session.  The  equivalence  of  the  CRT-1  and  CRT-3  conditions  aay  be  due  to  a 
tendency  for  exaalnees  to  Ignore  the  more  complex  features  of  the  CRT-3  condi¬ 
tion,  such  as  the  ability  to  return  to  earlier  paragraphs,  responding  Instead 
In  a  manner  similar  to  that  of  tha  CRT-1  format,  l.e.,  proceeding  straight 
through  the  test  ltaa  by  ltaa  without  backtracking.  The  lower  performance  for 
CRT-2  nay  have  resulted  from  the  increased  memory  requirement  caused  by  the 
particular  screen  configuration  used. 

The  reliability  analyses  found  that  the  PC  subtest  obtained  lower  lnter- 
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nal  consistencies  of  a  meaningful  magnitude  (>  *10)  whan  computer-adminis¬ 
tered.  However,  the  test-retest  reliabilities  whan  ooa  adainlatration  was  by 
eoaputar  vara  not  significantly  lower  than  tha  papa r-and-penc il-only  tast-re- 
test- reliabilities.  In  fact,  tha  Group  2  taat-ratast  corralatlona  warn  sig¬ 
nificantly  higher  (for  Modes  1  and  3)  than  those  froe  the  paper-and-pencll- 
only  group. 

Speeded  Subtests 

The  NO  and  CS  subtests,  chosen  on  the  basis  of  their  speeded  nature,  were 
adalnlstered  by  paper-end-pencil  and  by  two  CRT  conditions.  The  first  coaput- 
er  node  presented  one  ltaa  at  a  tins  on  the  screen,  with  each  Itaa  being 
erased  and  replaced  by  the  next  It an  following  the  examinee's  response.  The 
second  node  filled  the  screen  with  a  block  of  lteas,  with  the  examinee  re¬ 
sponding  to  the  first  Item  In  the  block.  This  Item  was  then  erased  and  the 
remaining  Items  In  the  block  shifted  upward  following  the  examinee's  response. 
The  number  of  Items  present  on  the  screen  decreased  In  this  manner  with  each 
response  until,  following  the  response  to  the  last  Item  In  the  block,  a  new 
block  of  lteas  appeared  on  the  screen. 

Although  the  Initial  analysis  revealed  significant  differences  between 
the  paper-and-pencll  and  CRT  media  for  both  the  NO  and  CS  subtests  In  Session 
1,  further  evaluation  demonstrated  equivalence  between  one  CRT  condition  and 
the  paper-and-pencll  medium  In  both  sessions.  For  the  NO  subtest,  this  equiv¬ 
alence  was  found  between  the  CRT-1  single- ltaa  node  and  the  paper-and-pencll 
condition,  with  the  CRT-2  group  scoring  significantly  lower.  For  CS,  the 
equivalent  conditions  were  the  CRT-2  multiple-item  screen  node  and  the  paper- 
and-pencll  condition,  with  the  CRT-1  single-item  group  scoring  much  higher 
than  either  of  the  others.  It  should  be  noted  that  for  every  comparison  be¬ 
tween  the  CS  CRT-2  and  paper-and-pencll  conditlona,  a  consistent  trend  of  mar¬ 
ginally  lower  CRT  scores  was  observed,  arguing  against  absolute  equivalence. 

For  both  speeded  subtests,  a  consistent  relationship  was  found  between 
tha  two  computer  conditions,  with  the  single-item  presentation  causing  higher 
scores  than  the  multiple-item  presentation  In  every  case,  the  multiple-item 
mode  perhaps  causing  distraction  or  confusion  for  the  examinees.  For  the  NO 
subtest,  this  resulted  In  highly  attenuated  scores  for  the  CRT-2  condition  as 
compared  with  paper-and-pencll  results.  For  the  CS  subtest,  the  single-item 
computer  node  provided  such  a  narked  Increase  in  performance  over  paper-and- 
pencll  administration  that  the  attenuating  affects  of  multiple-item  computer 
presentation  brought  scores  more  In  line  with  those  for  the  paper-and-pencll 
medium.  This  finding  may  be  deceptive,  however.  The  CS  single-item  presenta¬ 
tion  condition  may  actually  be  more  parallel  to  the  paper-and-pencll  perfor¬ 
mance,  differing  only  by  a  sealing  factor  dua  to  the  greater  response  speed 
afforded  the  CRT  examinee.  The  multiple-item  response  condition  nay  be  sub¬ 
ject  to  negative  effects  arising  from  rapid  response  tines  which  are  then  ne¬ 
gated  by  the  distracting  Influence  of  the  upward-shifting  multiple-item  screen 
format,  as  found  with  the  NO  subtest. 

The  test-retest  reliability  analysis  showed  for  both  NO  and  CS  that  ad¬ 
ministration  by  computer  In  one  session  significantly  lowered  the  correlation 
of  computer-administered  scores  with  those  from  paper-and-pencll  administra¬ 
tion,  In  comparison  to  the  paper-and-pencll  baseline  analysis.  This  suggests 
that  the  administration  of  speeded  tests  by  computer  might  result  In  test 
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•cons  with  somewhat  different  score  proport los  than  the  same  sposdod  tests 
administered  by  paper-and-pencil. 

Grephlcel  Subtests 


The  AS,  MC,  end  El  subtests,  chosen  for  their  graphical  end  standard  mul- 
tlple-cholce  text  content,  were  presented  In  a  single  CRT  node  In  addition  to 
paper-end- pencil  presentation.  The  fomat  of  the  computer  presentation  was 
Identical  to  that  of  the  paper-end-pencil  tests  with  digitized  graphical  Ima¬ 
ges,  copied  directly  from  the  ASVAB  test  booklets,  appearing  on  the  computer 
screen  with  the  appropriate  text. 

The  results  of  the  analyses  for  these  subtests  Indicated  e  straightfor¬ 
ward  equivalence  between  the  computer  end  paper-end-pencil  media  for  all  three 
•Obtests.  The  clarity  of  this  finding  can  be  attributed,  at  least  In  part,  to 
tha  construction  of  finely  detailed  computer  representations  of  all  graphical 
Images.  A  direct  test  of  the  equivalence  of  examinee  perception  of  these  ima¬ 
ges  was  provided  by  the  comparison  between  CRT  and  paper-and-pencll  conditions 
within  Session  1  for  the  MC  subtest,  the  content  of  which  Is  almost  entirely 
graphical.  This  comparison  Identified  no  differences  between  the  two  modes  of 
presentation,  thus  suggesting  equivalence. 

Subtest  Structure 


The  interrelationships  of  this  battery  of  subtests  as  a  whole  were  some¬ 
what  different  under  the  conditions  of  different  administration  methods.  The 
principal  factor  analyses  suggested  that  the  factor  structure  of  the  subtest 
•cores  for  computer-administered  Group  1  Session  1  contained  three  factors, 
whereas  only  two  factors  were  needed  to  explain  the  data  from  the  other  first 
session  paper-and-pencll  administrations.  The  LISREL  confirmatory  analysis 
supported  these  results  by  rejecting  a  model  of  the  equality  of  the  subtest 
covariance  matrices  whan  a  computer  administration  group  was  Included,  but  not 
rejecting  a  model  of  equal  covariance  matrices  when  only  covariance  matrices 
from  the  paper-end-pencil  groups  were  Included.  All  of  the  less  stringent 
tests  of  the  equality  of  factor  structure  models  across  administration  madia 
found  either  a  rejection  of  the  model  or  the  presence  of  non-positive  definite 
covariance  matrices.  A  finding  of  non-positive  definite  covariance  matrices 
is  an  indication  that  the  model  did  not  provide  a  suitable  fit  to  the  data. 

These  negative  LISREL  findings  could  be  due  to  the  slight  factor  struc¬ 
ture  difference  found  in  the  principal  factors  analyses  or  to  the  extreme 
power  of  the  LISREL  procedure  when  sample  sizes  and  degrees  of  freedom  are 
large.  The  model  of  equal  factor  structure  that  did  converge  to  meaningful 
parameter  estimates  produced  a  significant  chi-square,  but  the  degree  of  non- 
fit  was  quite  smell.  The  estimated  covariance  matrices  for  Groups  1,2,  and  3 
produced  only  zero,  one,  end  two  significant  normalized  residuals,  respective¬ 
ly.  Thus,  although  the  model  did  obtain  a  significant  chi-square  value,  it 
should  not  be  rejected  entirely. 


V.  CONCLUSIONS 
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The  results  of  the  present  study  suggest  that  attaining  equivalent  test 
results  between  computer  and  paper-and-pencil  administrations  is  feasible  for 
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the  power  subteats  of  the  ASVAB.  For  the  ease  of  etaaderd  multiple-choice 
text  lteaa  and  thoae  with  graphical  content ,  all  that  n ay  be  required  la  care¬ 
ful  development  of  touting  software,  with  adequate  attention  to  the  clarity 
end  detail  of  graphical  Image a* 

For  the  apeeded  aubteats,  the  preaent  findings  Indicate  that  the  mode  of 
screen  presentation  of  test  lteaa  can  drastically  Influence  the  level  of  exaa- 
lnee  perforuance.  For  the  MO  subtest,  an  Indication  of  equivalence  between 
the  single-item  CRT  and  paper-and-pencll  presentation  conditions  was  estab¬ 
lished,  For  CS,  however,  although  the  aultiple-itea  screen  condition  produced 
alnllar  scores  to  thoae  fron  the  paper-end-pencil  aediua,  the  tread  over  mul- 
tiple  conparisons  between  the  two  showed  that  the  coaputer  scores  were  aargin- 
ally  lower  in  every  case.  This  finding  suggests  that  further  research  on  al¬ 
ternative  screen  configurations  and  enlarged  coaputer  Instruction  and  practice 
seta  nay  be  appropriate  to  bring  computer  perforasaca  wore  in  line  with  Its 
paper-and-pencll  counterpart.  Continued  Investigation  aay  also  reveal  that 
the  higher- scoring  slngle-itaa  screen  condition  Is  actually  aore  consistent 
with  paper-and-pencll  performance,  and  that  a  scaling  factor  Is  required  to 
compensate  for  the  fester  response  time  made  possible  by  the  substitution  of  a 
computer  keyboard  for  a  paper-and-pencll  answer  sheet. 

The  PC  subtest  offers  perhaps  the  greatest  challenge  to  equivalence  in 
the  ASVAB  battery.  For  this  subtast,  the  present  results  deaonstrate  the  sen¬ 
sitivity  of  examinee  performance  to  alternative  screen  presentation  mods  for 
this  rather  coaplaX  test.  The  findings  support,  however,  the  benefits  of 
pre-test  practice,  simplicity  of  screen  format  design,  and  detailed  Instruc¬ 
tion  sets  In  equalising  computer  and  paper-and-pencll  performance.  Recommen¬ 
dations  for  further  research  Include  experimentation  with  varying  numbers  of 
practice  Items,  alternative  screen  formats,  Instruction  sets  emphasizing  spe¬ 
cific  aspects  of  the  computerized  PC  subtest,  and  administration  of  the  PC 
subtest  after  administration  of  other  subtests  In  the  ASVAB  battery  to  allow 
greater  examinee  familiarity  with  the  coaputer  before  the  presentation  of  the 
PC  Items. 
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■for  Session  1  papar-and-pancll  data,  tha  nods  designation  rafara  to  which  eonpotar  adnlnistration  nod*  followed 
the  papar-and-pancil  adnlnistration  (Group  2);  for  Session  2  this  designation  indicates  that  tha  papar-and-pancll 
data  war*  calculated  fron  tests  that  followad  tha  particular  counter  adnlnistration  node  listad  (Cronp  1). 
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